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DEFINITIONS 

IDA  publishes  the  following  documents  to  report  the  results  of  its  work. 


Reports 

Reports  are  the  most  authoritative  and  most  carefully  considered  products  IDA  publishes. 
They  normally  embody  results  of  major  projects  which  (a)  have  a  direct  bearing  on 
decisions  affecting  major  programs,  (b)  address  issues  of  significant  concern  to  the 
Executive  Branch,  the  Congress  and/or  the  public,  or  (c)  address  issues  that  have 
significant  economic  implications.  IDA  Reports  are  reviewed  by  outside  panels  of  experts 
to  ensure  their  high  quality  and  relevance  to  the  problems  studied,  and  they  are  released 
by  the  President  of  IDA. 

Group  Reports 

Group  Reports  record  the  findings  and  results  of  IDA  established  working  groups  and 
panels  composed  of  senior  individuals  addressing  major  issues  which  otherwise  would  be 
the  subject  of  an  IDA  Report.  IDA  Group  Reports  are  reviewed  by  the  senior  individuals 
responsible  for  the  project  and  others  as  selected  by  IDA  to  ensure  their  high  quality  and 
relevance  to  the  problems  studied,  and  are  released  by  the  President  of  IDA. 


Papers 

Papers,  also  authoritative  and  carefully  considered  products  of  IDA,  address  studies  that 
are  narrower  in  scope  than  those  covered  in  Reports.  IDA  Papers  are  reviewed  to  ensure 
that  they  meet  the  high  standards  expected  of  refereed  papers  in  professional  journals  or 
formal  Agency  reports. 

Documents 

IDA  Documents  are  used  for  the  convenience  of  the  sponsors  or  the  analysts  (a)  to  record 
substantive  work  done  in  quick  reaction  studies,  (b)  to  record  the  proceedings  of 
conferences  and  meetings,  (c)  to  make  available  preliminary  and  tentative  results  of 
analyses,  (d)  to  record  data  developed  in  the  course  of  an  investigation,  or  (e)  to  forward 
information  that  is  essentially  unanalyzed  and  unevaiuated.  The  review  of  IDA  Documents 
is  suited  to  their  content  and  intended  use. 
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INSTITUTE  FOR  DEFENSE  ANALYSES 

IDA  Central  Research  Program 


PREFACE 


This  volume  was  prepared  from  a  verbatim  transcript  recorded  at  an  IDA  Round 
Table  held  on  April  27,  1995,  and  organized  according  to  major  themes  and  topics.  In 
some  cases  it  was  felt  that  the  sense  of  the  participants  was  best  presented  in  their  own 
vernacular,  while  in  other  cases  concepts  were  clarified  or  pared  by  the  editors.  Every 
effort  was  made  to  retain  the  substantive  content  of  the  Round  Table  discussions,  and  to 
this  end  all  participants  were  afforded  an  opportunity  to  review  a  final  draft  of  the 
document  prior  to  its  publication. 

Review  comments  of  an  editorial  or  stylistic  nature  were  generally  incorporated 
into  the  document’s  text.  Comments  regarding  the  substance  of  the  text  or  which  would 
have  altered  the  record  are  handled  as  footnotes  to  the  text  and  marked  as  “Review  Note.” 
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INTRODUCTION 


On  April  27,  1995,  the  Institute  for  Defense  Analyses  (IDA)  conducted  a  Round 
Table  discussion  entitled  “Assessing  the  Economic  and  National  Security  Benefits  of 
Publicly  Funded  Technology  Investments.”  This  meeting  was  held  to  explore  two  areas: 

(1)  practical  issues  that  government  agencies  should  consider  in  selecting 
technology  investments,  and 

(2)  criteria  for  assessing  or  measuring  the  success  of  selected  investments. 

The  co-chairs  of  the  Round  Table  were  Joseph  Stiglitz,  chair  of  the  President’s 
Council  of  Economic  Advisors  (CEA);  John  Gibbons,  director  of  the  Office  of  Science 
and  Technology  Policy  (OSTP);  and  Christopher  Jehn,  director  of  IDA’s  Strategy, 
Forces,  and  Resources  Division. 

This  document  sununarizes  and  highlights  the  main  discussions  of  the  day.  Dr. 
Gibbons  first  established  the  focus  of  the  Round  Table  in  his  keynote  address.  This  was 
followed  by  three  sessions  combining  presentations  and  discussions  by  Round  Table 
participants  that  dealt  in  detail  with  a  set  of  current  issues  concerning  the  appropriate 
methods  of  establishing  objectives  and  measuring  the  outcomes  of  national  investment  in 
new  technologies.  Those  sessions  were: 

(1)  Decision  Criteria  for  Technology  Investments:  Understanding  the  Rationales 
for  Commercial  and  Dual-use  Technology  Programs 

(2)  Tailoring  Metrics  to  Economic  Policy  Objectives:  The  Case  of  ATP,  MEP, 
and  the  NIST  Laboratories 

(3)  Metrics  for  Multiple  Policy  Objectives:  The  Case  of  TRP  and  Dual-use. 

Economic  studies  have  consistently  shown  that  the  private  and  social  returns  on 
investments  in  R&D  are  very  high — indeed,  much  higher  than  returns  on  investments  in 
other  areas.  Estimating  expected  returns  on  science  and  technology  investments  in 
advance,  or  ex  ante,  enables  decision  makers  to  make  wise  use  of  limited  public 
resources.  One  role  of  metrics,  therefore,  is  to  assist  decision  makers  by  providing 
orderly  information  on  the  likely  impact  of  proposed  investments,  based  upon  historical 
observation  and  experience. 
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Metrics  are  also  an  important  part  of  designing  good  incentive  structures.  Today 
we  focus  increasingly  on  measured  or  anticipated  performance  in  designing  programs  and 
selecting  areas  for  investment.  If  one  can  measure  performance,  then  one  can  relate 
rewards  to  performance.  This  is  a  key  to  developing  efficient  and  just  reward  systems. 
At  the  same  time,  basing  rewards  on  inappropriate  or  inaccurate  measures  of  performance 
may  lead  to  more  harm  than  good. 

Metrics  also  support  retrospective,  or  ex  post  evaluations  of  investments  and 
provide  a  basis  for  government  decision  makers  to  evaluate  and  discuss  the  benefits  of 
investments  in  science  and  technology.  Case  studies  are  particularly  effective  in  this 
regard.  To  be  effective,  however,  such  case  studies  must  relate  to  the  everyday 
experiences  of  the  lay  public,  so  that  they  can  know  how  the  fruits  of  technology 
investment  affect  their  lives,  from  home  entertainment  to  surgery.  Such  case  studies  are 
probably  the  most  forceful  way  of  educating  Congress  and  the  American  people  about  the 
value  of  S&T  investments. 

Dr.  Gibbons  established  in  his  keynote  address  that  the  government  needs  to  agree 
on  valid  metrics  to  effectively  assess  technology  investments  by  the  public  sector.  Round 
Table  participants  discussed  three  functions  of  metrics  as  they  relate  to  the  economic 
assessment  of  the  government’s  technology  investments.  This  document  is  meant  to 
serve  as  a  summary  record  of  those  discussions. 

BACKGROUND 

A  transition  in  federal  science  and  technology  (S«&T)  policy  is  occurring  in  the 
dynamic  political  and  economic  environment  created  by  the  end  of  the  Cold  War  and  the 
simultaneous  intensification  of  global  economic  competition.  Investments  in  applied 
technologies  are  an  important  part  of  the  current  administration’s  efforts  to  foster  long¬ 
term  economic  growth  and  national  security.  Current  administration  policy  emphasizes 
explicit  links  between  federal  government  technology  programs  and  the  goals  and 
priorities  of  U.S.  industry.  Government-industry  partnerships  are  understood  to  be 
increasingly  important  in  strengthening  America’s  global  industrial  competitiveness. 

During  the  preceding  half-century,  technology  advanced,  in  part,  through  the 
stimulus  of  government  supported  basic  science.'  Examples  of  this  process  may  be 
found  in  the  technological  fruits  of  space-  and  military-related  research  conducted  by  the 


'  Vannevar  Bush  wrote  of  “government-supported  basic  science”  in,  Science,  the  Endless  Frontier. 
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National  Aeronautics  and  Space  Administration  (NASA)  and  the  Department  of  Defense 
(DoD).2  Through  the  mid-1970s,  commercial  exploitation  of  the  results  of  such  federally 
funded  R&D  contributed  to  the  United  States’  position  as  world  leader  in,  science, 
national  security,  and  economic  competitiveness.  The  United  States  used  its  premier 
position  to  promote  open  trade  and  global  economic  development  to  the  benefit  of  both 
its  allies  and  itself.  In  the  process,  it  also  assisted  in  creating  sophisticated  economic 
competitors,  the  most  successful  of  whom,  many  believe,  use  government  policy  more 
purposefully  than  has  the  U.S.,  to  leverage  science  and  technology  for  the  advancement 
of  their  national  welfare. 

In  the  face  of  such  foreign  competition,  policy  makers  in  the  United  States  today 
assign  a  higher  priority  to  linking  the  technology  activities  of  government  and  industry. 
At  the  same  time,  the  end  of  the  Cold  War  has  profoundly  altered  the  political  landscape, 
changing  the  basis  and  the  rationale  by  which  government  technology  investments  are 
made.  The  opportunities  for  commercial  “spill-overs”  to  the  commercial  sector  from 
defense-related  research  have  diminished,  so  that  today  policies  favoring  dual-use  and 
“spin-on”  technologies  appear  to  offer  the  greatest  promise  for  simultaneously  meeting 
the  goals  of  national  security  while  stimulating  general  economic  growth  through 
technological  innovation.  With  highly  constrained  budgets,  however,  such  technology 
investments  must  be  based  on  a  clear  prospect  of  tangible  benefits  that  can  be  monitored 
and  measured  effectively. 

In  recognition  of  this  need  to  carefully  select  technology  investment  programs 
based  on  measurable  results,  the  Round  Table  was  convened  to  explore  three  key  issues: 

•  How  can  agencies  with  applied  technology  programs  best  select  where  to 
invest  public  resources? 

•  How  should  the  government  seek  to  measure  the  results  of  its  investments  in 
technology  development  and  application? 

•  What  lessons  have  been  learned  about  managing  the  intersection  of  economic 
growth,  national  security,  and  the  pursuit  of  other  national  objectives? 


2  Review  Note:  This  research  included  scientific  research  supported  by  the  National  Science 
Foundation  and  the  National  Institutes  of  Health,  as  well  as  government  support  of  applied  research 
and  technology  development  for  military,  space,  and  health-related  purposes.  Such  investments 
emiched  the  civilian  economy  through  spin-offs  and  technological  spillovers,  especially  in  the  areas  of 
microelectronics,  computer  hardware  and  software,  and  pharmaceuticals  and  biotechnology. 
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The  Round  Table  reviewed  and  sought  to  improve  ongoing  national  efforts  to 
create  a  framework  for  assessing  and  measuring  the  results  of  national  technology 
investments.  Participants  included  many  authoritative  observers  of  the  economics  of 
technological  innovation.  They  represented  the  perspectives  of  academia,  industry,  and 
the  federal  government.  Discussions  centered  on  different  types  of  quantitative  and 
qualitative  measures  and  their  applicability  to  different  kinds  of  government  technology 
investments.  Within  this  framework,  participants  explored  potential  linkages  between 
research,  innovation,  and  long-term  economic  growth. 

OVERVIEW  OF  ROUND  TABLE  PROCEEDINGS 

Keynote  Address 

Dr.  John  Gibbons  delivered  the  keynote  address.  He  first  described  the  political 
and  economic  factors  that  affect  science  and  technology  policy,  and  then  discussed  the 
importance  of  government  support  in  this  area.  Dr.  Gibbons  noted  that  many  in  Congress 
question  whether  the  government  should  be  investing  in  certain  types  of  science  and 
technology  they  regard  as  outside  the  province  of  government  and  are,  in  fact,  seeking  to 
cut  many  types  of  federal  science  and  technology  (S&T)  funding.  In  the  face  of  such 
questions,  the  Clinton  administration  is  not  only  reaffirming  the  government’s  historical 
support  of  fundamental  science,  but  is  seeking  partnerships  with  the  private  sector  for 
certain  kinds  of  applied  research.  The  administration  recognizes  that  it  must  have  clear 
measures,  or  metrics,  for  choosing  investment  programs,  for  measuring  the  benefits  of 
those  programs,  and  for  demonstrating  those  benefits  to  the  American  taxpayer. 

Session  One:  Decision  Criteria  for  Technology  Investments:  Understanding  the 
Rationales  for  Commercial  and  Dual-Use  Technology  Programs 

The  first  Round  Table  session  sought  to  identify  indicators  that  may  be  used  to 
help  government  decision  makers  target  publicly  financed  investments  in  ways  which  can 
generate  high  social  rates  of  return.  The  participants  considered  both  ex  ante  criteria  for 
technology  investments  and  ex  post  methods  of  evaluation.  The  discussion  addressed 
several  questions,  including:  What  are  the  best  criteria  for  choosing  public  technology 
investments?  How  can  we  identify  federal  S&T  investments  with  a  high  probability  of 
resulting  in  commercially  viable  technologies?  What  can  we  learn  from  the  experience 
of  other  nations,  states,  and  private  industry  about  identifying  areas  for  investment  with  a 
high  potential  for  social  returns?  And,  finally,  how  can  performance  measures  be  used. 
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along  with  sunset  provisions  and  other  limits  on  investment,  to  tell  government  decision 
makers  when  to  terminate  public  investments? 

Session  Two:  Tailoring  Metrics  to  Economic  Policy  Objectives:  The  Case  of  ATP, 
MEP,  and  the  NIST  Laboratories 

The  second  session  drew  upon  the  programs  and  experience  of  the  National 
Institute  of  Standards  and  Technology  (NIST),  within  the  Department  of  Commerce,  as  a 
surrogate  for  all  commercial  technology  programs.  The  goal  was  to  identify  the  strengths 
and  weaknesses  of  the  tools  available  for  assessing  public  technology  investments,  and  to 
determine  how  to  apply  different  sets  of  performance  metrics  to  different  types  of 
commercial  technology  projects.  Questions  addressed  included:  What  do  we  know,  and 
when  do  we  know  it?  What  sorts  of  impacts  are  we  trying  to  measure?  Are  these  the 
right  ones?  What  are  the  uses  and  limitations  of  the  quantitative  and  qualitative  measures 
we  currently  employ?  When  are  quantitative  measures  inappropriate  and  misleading,  and 
how  can  qualitative  approaches  be  made  more  rigorous  and  broadly  applicable?  Given 
that  the  outcomes  only  become  apparent  in  the  long  term,  what  sorts  of  short  term  and 
interim  measures  are  appropriate? 

Session  Three:  Metrics  for  Multiple  Policy  Objectives:  The  Case  of  TRP  and  Dual- 
Use 

The  third  Round  Table  session  used  the  Defense  Department’s  Technology 
Reinvestment  Project  (TRP)  as  an  archetype  for  understanding  the  issues  to  be  considered 
when  making  dual-use  investments  to  benefit  national  security  by  leveraging  the 
commercial  industrial  base  for  defense  applications.  It  addressed  the  following  issues: 
What  is  a  dual-use  technology?  What  is  the  role  of  dual-use  programs  in  promoting  a 
more  robust  national  security?  How  can  national  security  benefit  from  dual-use 
technology  investments  and  an  integrated  commercial  and  military  industrial  base?  How 
could  such  integration  lead  to  more  affordable  weapons  systems?  What  do  we  know 
about  simultaneously  managing  defense  needs,  commercial  environments,  and  economic 
growth?  Are  there  lessons  to  be  learned  from  dual-use  that  will  benefit  public 
investments  for  other  national  objectives  such  as  health,  energy,  the  environment,  and 
education? 

The  following  sections  of  this  report  provide  detailed  descriptions  of  the  Round 
Table  proceedings. 
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SUMMARY  OF  KEYNOTE  ADDRESS 


Dr.  John  Gibbons 
Assistant  to  the  President 
for  Science  and  Technology,  and 
Director,  Office  of  Science  and  Technology  Policy 

Dr.  Gibbons  began  by  noting  the  timeliness  of  the  Round  Table,  given  the  current 
debate  in  Congress  about  the  role  of  government  in  American  life  and,  indeed,  of  public 
investments  in  science  and  technology.  In  his  view,  many  conservative  Democrats  and 
Republicans  alike  see  the  government  as  obtrusive  and  accordingly  are  focusing,  almost 
single-mindedly,  on  removing  government  from  our  lives.  The  concomitant  goal  of 
balancing  the  budget  in  a  very  short  time  only  lends  impetus  to  the  push  to  drastically 
reduce  government  support  of  science  and  technology. 

The  Clinton  administration  has,  from  the  outset,  supported  deficit  reduction  and 
has  made  great  strides  in  that  regard.  Gibbons  explained.  But  the  administration  is  also 
seeking  to  maintain  an  investment  strategy  that  complements  deficit  reduction,  in  order  to 
create  the  potential  for  returns  on  investment  that  more  than  justify  controlled  deficit 
spending. 

There  are  those  in  Congress  who  would  draw  a  distinction  between  science  and 
technology,  and  support  federal  assistance  to  basic  science  while  withdrawing  support 
from  technology  development.  Dr.  Gibbons  stressed  that  sharp  distinction  between  the 
two  domains  are  not  supportable,  each  builds  constantly  on  the  gains  of  the  other. 
Moreover,  Dr.  Gibbons  noted,  some  budget  reduction  proposals  include  dramatic  funding 
reductions  not  only  for  technology,  but  also  for  science.  For  example,  about  $200  million 
of  the  $1  billion  that  Congress  has  proposed  rescinding  from  the  1995  budget  comes  from 
scientific  programs.  Half  of  that  is  from  the  Department  of  Defense’s  support  of 
computer  science,  math  research,  and  engineering  education  at  American  universities. 
Neal  Lane,  the  director  of  the  National  Science  Foundation,  has  been  told  to  plan  for  a 
budget  cut  of  as  much  as  a  20  percent  in  the  years  ahead  by  the  Chairman  of  the  House 
Budget  Committee,  Representative  Kasik. 
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The  administration,  in  contrast,  wants  to  continue  the  federal  government’s 
historic  support  of  fundamental  science,  as  well  as  its  recent  support  for  applied  science 
and  engineering.  Given  the  political  and  economic  realities  of  our  time,  it  is  also 
concentrating  on  building  partnerships  with  industry,  universities  and  community 
colleges,  and  state  and  local  governments  to  develop  technologies  that  offer  substantial 
benefits  to  the  economy  as  a  whole.  Many  of  these  activities  are  in  jeopardy,  according 
to  Dr.  Gibbons,  who  cited  the  following  examples: 

•  The  Advanced  Technology  Program  (ATP).  This  program  is  run  by  the 
National  Institute  of  Standards  and  Technology  (NIST).  It  is  industry-led, 
cost-shared,  and  projects  are  selected  competitively  strictly  on  the  basis  of 
merit. 

•  The  partnership  for  a  new  generation  of  vehicles,  also  known  as  the  Clean 
Car  Program.  The  goal  is  to  develop,  within  10  years  or  less,  a  new  vehicle 
that  gets  three  times  the  mileage  currently  attainable — without  sacrificing 
performance,  quality,  safety,  and  affordability. 

•  The  Environmental  Technology  Strategy.  This  recently  released  strategy 
calls  for  government-industry  partnerships  for  innovation  in  the  U.S. 
environmental  technology  industry  both  to  clean  up  pollution  from  the  past 
and  also  to  prevent  pollution  in  the  future.  This  strategy  focuses  both  on 
fulfilling  domestic  needs  and  on  exporting  into  a  rapidly  growing  world 
market. 

•  The  NIST  manufacturing  extension  partnership.  This  program  is  a  grass 
roots  network  of  industrial  extension  services  that  can  help  small 
manufacturing  firms  compete  more  effectively  with  modem  manufacturing 
equipment  and  practices. 

•  The  DoD  Technology  Reinvestment  Project  (TRP).  This  program  is  designed 
to  encourage  the  commercial  development  of  technologies  that  are  also 
critical  to  our  military.  Low-cost  commercial  manufacturing  practices  and 
economies  of  scale  resulting  from  commercial  production  would  make  these 
technologies  much  more  affordable  for  military  users. 

•  The  National  Information  Infrastructure.  This  program  supports  R&D  that 
would  continue  our  nation’s  leadership  in  high-performance  computing,  and 
improves  the  accessibility  of  that  technology. 

•  The  Climate  Change  Action  Program.  This  program  promotes,  among  other 
objectives,  energy  efficiency  and  the  use  of  renewable  energy  to  conserve 
resources  and  limit  the  impact  of  human  activities  on  the  global  climate. 

•  Industry-led  Cooperative  Research  and  Development  Agreements  ( CRADAs). 
These  agreements,  which  have  multiplied  tenfold  since  the  beginning  of  the 


8 


administration,  help  move  important  new  developments  toward  the  commer¬ 
cial  marketplace,  with  benefits  in  such  areas  as  health  care,  transportation, 
and  home  design. 

Dr.  Gibbons  stressed  that  programs  such  as  these  have  been  successful  both  in  the 
United  States  and  in  other  countries,  but  that  the  public  and  the  Congress  know  little 
about  them.  Thus  he  suggested  that  the  participants  educate  the  public  and  Congress 
about  such  programs  in  numerous  and  varied  forums. 

Dr.  Gibbons  also  stressed  that  in  supporting  these  programs  the  government  must 
answer  clearly  the  following  questions: 

•  Why  is  government  investment  in  technologies  that  benefit  us  all 
indispensable? 

•  What  has  changed  in  the  world  to  make  these  investments  more  important 
than  ever  before? 

•  What  has  been  happening  in  the  world  economy  that  now  underscores  the 
imperative  for  us  to  join  other  nations  that  have  successfully  joined  public 
and  private  interests? 

•  How  can  we  know  whether  we  are  really  getting  the  results  we  expect  and 
desire  from  our  publicly  funded  technology  investments? 

•  Can  we  devise  measures  to  guide  our  decision  making  as  we  select  projects, 
monitor  their  progress,  and  evaluate  their  ultimate  success  or  failure? 

In  concluding.  Dr.  Gibbons  focused  on  the  need  to  identify  ways  to  measure  the 
success  of  government-supported  technology  programs.  He  cited,  as  an  example,  recent 
research  at  Georgia  Tech  which  evaluated  the  experiences  of  more  than  200  firms  that 
had  cooperated  with  federal  laboratories  to  develop  new  technologies  in  a  recent  5-year 
period. 

•  89  percent  said  that  the  interaction  was  a  good  use  of  company  resources  and 
that  the  interaction  yielded  a  net  benefit  to  the  company. 

•  22  percent  of  the  cooperating  firms  already  had  a  new  product  or  a  process  or 
service  as  a  result  of  the  interaction. 

•  38  percent  more  had  a  new  product  under  development,  and  another  23 
percent  had  improved  an  existing  product. 

Dr.  Gibbons  cited  job  creation  as  another  possible  measure  but  noted  that  the 
Georgia  Tech  research  had  not  indicated  direct  success  in  this  area.  He  encouraged 
Round  Table  participants  to  share  their  own  experiences  and  knowledge  in  discussing 
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how  to  select  technologies  that  are  worthy  of  government  investment,  how  to  measure  the 
results,  and  how  to  anticipate  outcomes. 
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Session  One 

DECISION  CRITERIA  FOR  TECHNOLOGY  INVESTMENTS: 
UNDERSTANDING  THE  RATIONALES  FOR  COMMERCIAL 
AND  DUAL-USE  TECHNOLOGY  PROGRAMS 


Session  one  addressed  ex  ante  criteria  for  choosing  technology  investments,  and 
ex  post  methods  of  evaluating  them.  This  section  first  summarizes  the  main  points  of 
presentations  given  by  government  and  industry  participants,  and  then  highlights  key 
perspectives  that  emerged  from  the  Round  Table  discussion. 

EX  ANTE  CRITERIA  FOR  TECHNOLOGY  INVESTMENT 

Science  and  technology  policy  comprises  a  key  part  of  the  Clinton 
administration’s  platform  for  economic  growth  and  industrial  competitiveness.  The 
recent  shift  in  the  composition  of  Congress,  however,  has  resulted  in  a  broad-ranging 
debate  about  the  appropriate  roles  of  government — a  debate  that  is  likely  to  continue  over 
the  next  year  and  a  half.  During  that  time,  national  science  and  technology  policy  is  sure 
to  remain  an  issue.  The  Clinton  administration  will  most  likely  argue,  as  do  most 
economists,  that  there  are  particular  areas  in  science  and  technology  where  national 
government  clearly  has  a  role,  especially  where  the  government  invests  in  technology  for 
the  public  good,  and  where  technology  investments  for  its  own  missions  work  to  improve 
the  national  technology  infrastructure,  or  provide  knowledge  which  can  be  exploited  by 
the  commercial  sector. 

Perhaps  the  critical  issue  to  be  resolved  in  the  anticipated  debate  is  represented  by 
the  phrase  “picking  winners  and  losers.”  This  phrase  has  been  used  to  connote  a  process 
of  advocacy  by  selective  investment  that  critics  believe  government  should  avoid.  But 
historically  this  process  has  been  unavoidable  in  the  formulation  of  public  policy. 
Government  chooses  winners  and  losers  every  day  by  virtue  of  its  routine  operations  in 
making  tax  and  regulatory  policies,  in  selecting  key  mission  areas,  and  in  making  R&D 
decisions.  Arguably,  “picking  winners  and  losers”  is  not  only  unavoidable,  but  essential 
to  good  government.  If  so,  the  issue  is  not  whether  or  not  the  government  should  make 
choices,  but  how  to  make  those  choices  good  ones.  The  bases  for  making  sound  choices 


11 


in  science  and  technology  investments,  termed  ex  ante  decision  criteria  for  the  purposes 
of  the  Round  Table,  are  not  always  apparent,  and  can  be  extremely  complex. 

Policy  Considerations 

The  decisionmaking  processes  at  the  policy  and  program  levels  are  distinct  but 
quite  related.  The  policy  level  is  concerned  with  the  national  technology  system.  In 
total,  among  all  government  and  privates  sources,  the  United  States  spends  approximately 
$160  billion  a  year^  on  research  and  development.  To  best  accomplish  the  nation’s 
various  science  and  technology  goals,  decision  makers  try  to  allocate  these  investments 
across  a  host  of  different  missions  in  interrelated  program  areas  within  the  broad  science 
and  technology  community.  At  this  level,  ex  ante  decision  tools  are  aimed  at  improving 
the  mix  of  investments  in  science  and  technology  programs 

At  the  program  level  the  issues  of  ex  ante  selection  are  concerned  with  the 
allocation  of  resources  to  particular  competing  approaches.  In  particular,  among  many 
competing  approaches  to  solving  a  problem,  or  among  many  promising  technology 
applications,  how  should  one  choose,  in  advance,  those  to  support  with  federal  funds? 

Economic  Considerations 

In  the  last  decade  the  commercial  marketplace  has  become  an  extremely 
aggressive  environment.  The  United  States  has  lost  its  leadership  in  various  industries 
and  is  faced  with  threats  to  its  leadership  in  numerous  others,  although  there  remain  many 
very  healthy  and  competitive  industries  in  the  United  States. 

As  one  important  consequence  of  this  situation,  companies  have  been  reducing 
funding  for  long-term  research  and  focusing  on  short-term  development^  Such  changes 
can  be  found  in  many  large  conunercial  laboratories,  such  as  those  at  AT&T,  General 
Electric,  and  DuPont.  Almost  without  exception,  such  organizations  today  are  much 
narrower  in  their  investments,  more  focused  on  the  company’s  business,  and  more  closely 
linked  to  the  firm’s  operating  divisions.  Industry  is  focusing  its  R&D  investments  almost 
exclusively^  on  products  and  processes  that  are  appropriable  to  individual  companies  and 


3  Review  Note:  The  annual  expenditures  in  the  U.S.  from  all  sources  on  R&D  are  closer  to  $1 80  billion 
per  annum. 

^  Review  Note :  This  comment  and  the  preceding  paragraph  make  the  United  States  appear  as  if  it  is  on 
its  last  legs  in  the  R&D  arena,  an  implication  that  is  patently  not  true. 

^  Review  Note:  Perhaps  this  should  be  “more  narrowly”  rather  than  almost  exclusively. 
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are  therefore  expected  to  provide  a  return  on  investment  to  the  firm  which  will  help 
maintain  or  regain  its  competitiveness. 

Government,  on  the  other  hand,  has  traditionally  invested  in  the  general  science 
and  engineering  base  at  universities  on  the  one  hand,  and  in  mission-related  technologies 
on  the  other.  Government  decision  makers  have  believed  that  the  results  of  such  public 
sector  investments  would  inevitably  “spin  off’  to  industry  and  would  thus  provide  a 
resource  for  commercial  purposes.  Therefore,  in  the  U.S.  there  has  never  been  a 
purposeful,  technology  investment  strategy  to  promote  the  general  economic  welfare, 
national  economic  growth,  and  international  competitiveness  of  American  firms.^ 

In  today’s  world  such  a  laissez-faire  approach  to  linking  public  sector  investment 
to  general  economic  benefit  is  simply  not  enough.  It  has  become  very  important,  at  the 
policy  level,  to  rethink  federal  R&D  allocations  as  part  of  a  strategy  that  attempts  to 
balance  traditional  mission  investments  in  defense,  health,  energy,  and  space  with 
national  technology  needs  in  the  general  economy.  This  balance  should  continue  to 
include  a  commitment  to  funding  basic  science  and  engineering  research  in  the  university 
base. 

Toward  that  end,  the  Advanced  Technology  Program  (ATP),  administered  by  the 
National  Institute  of  Standards  and  Technology  (NIST),  is  targeting  commercial 
infrastructure  technologies  that  are  long-term  and  high  risk,  and  that  firms  are  unlikely  to 
undertake  themselves  because  of  issues  of  appropriability  to  the  firm  and  uncertainty  of 
result.  ATP  investments  end  prior  to  the  point  of  product  development.  Firms  must 
themselves  fund  production,  manufacturing,  sales,  marketing,  distribution,  and  advertis¬ 
ing.  The  ATP  targets  investments  toward  overcoming  fundamental  technical  barriers  so 
that  the  marketplace  can  work  more  efficiently. 

Selection  criteria  for  ATP  projects  include  such  factors  as  the  technical  soundness 
of  a  proposal  and  its  attractiveness  to  the  R«&D  community,  the  potential  for  the  proposed 
project  to  provide  broad-based  economic  benefits  for  the  U.S.,  consideration  of  the  ability 
of  a  proposal  to  take  a  technology  downstream  into  the  marketplace,  and  the  level  of 
commitment,  experience,  and  qualifications  of  the  proposal  team. 


^  Review  Note:  The  United  States  did  invest  in  breeder  reactors  and  supersonic  transports,  so  there  have 
been  purposeful  technology  investment  strategies  in  the  past,  contrary  to  this  assertion. 
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ATP  funds  are  allocated  on  a  strategic  investment  basis,  whereby  approximately 
$150  million  is  allocated  over  a  5-year  period  for  each  technology  focus  area.  Strategic 
areas  are  chosen  based  upon  advice  from  industry  and  a  consideration  of  the  following 
types  of  issues: 

•  What  is  the  economic  potential  of  funding  a  particular  technology  area? 

•  Where  are  the  good  technical  ideas? 

•  Where  is  the  commitment  from  the  industry,  not  just  to  share  the  cost  of  the 
program,  but  to  take  it  downstream? 

•  Where  are  the  places  where  federal  investment  will  have  the  highest 
leverage? 

•  How  do  we  ensure  that  we  do  not  duplicate  private  sector  investments? 

•  How  do  we  ensure  duplication  does  not  occur  with  other  agencies  around  the 
government? 

Because  the  objective  of  the  ATP  is  to  achieve  national  economic  goals,  and 
because  that  requires  industry  to  take  technology  to  market,  the  role  of  industry  is  central 
to  the  ATP  technology  area  selection  process.  Every  area  funded  begins  with  a  concept 
in  a  company  or  in  a  group  of  companies,  not  with  the  government. 

Requirements  of  Military  End-Users 

In  contrast,  the  Department  of  Defense  invests  in  dual-use  technologies  based 
upon  a  clear  and  compelling  need  as  judged  by  military  end-users.  In  choosing  projects 
for  investment,  defense  decision  makers  assess  the  state  of  the  relevant  technology,  the 
likelihood  that  an  investment  will  truly  accelerate  the  pace  of  maturation,  and  the  impact 
on  defense  of  the  new  capabilities  that  may  be  achieved.  In  addition,  defense  technology 
investment  decisions  are  based  on  an  in-depth  understanding  and  characterization  of  the 
supplier  industry.  What  is  the  inherent  industrial  capability  to  make  certain  investments? 
What  resources  are  required  to  make  a  difference  or  to  achieve  specific  goals?  What 
investments  will  suppliers  most  likely  make,  and  where  will  they  produce  the  resulting 
product?  The  decision  to  produce  offshore  or  onshore  has  significant  national  security 
implications. 

The  Potential  for  Government-Industry  Partnership 

There  is  a  strong  consensus  that  government  has  played  a  key  role  in  developing 
microelectronics  and  will  probably  continue  to  do  so  as  the  technology  evolves  and  new 
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inventions  are  created.  Although  investments  in  microelectronics  research  and 
development  originated  in  industry,  a  great  deal  of  critical  R&D  was  sponsored  by 
government.  Originally  termed  “molecular  electronics,”  this  technology  emerged  in  the 
1960s,  as  the  result  of  an  effective  government  partnership  with  industry. 

In  the  1980s,  a  major  shift  to  offshore  microelectronics  production,  a  concern  for 
profits,  and  a  regeneration  of  microelectronics  technologies  led  to  a  renewed  government 
commitment  to  bolster  the  nation’s  competitive  position.  This  was  expressed  in  a  variety 
of  programs,  not  all  of  which  were  completely  successful.  SEMATECH  (Semiconductor 
Manufacturing)  is  an  example  of  success,  while  the  VHSIC  (Very  High  Speed  Integrated 
Circuits)  program  provides  important  lessons  regarding  the  need  to  carefully  target 
government  funds  to  intended  recipients.^  More  recent  efforts  in  federal  support  to 
microelectronics,  particularly  in  the  area  of  flat  panel  displays,  are  too  embryonic  to 
judge. 

SEMATECH.  Some  argue  whether  it  was  a  correct  decision  to  proceed  with 
SEMATECH,  and  others  see  the  venture  as  an  example  of  government  making  sound 
technology  investments  in  the  microelectronics  business,  which  is  the  underpinning  of  the 
information  age.  The  original  object  of  this  program  was  to  improve  the  U.S. 
manufacturing  capabilities  for  a  specific  class  of  semiconductor  firms,  with  some  focus 
on  providing  the  capabilities  to  firms  that  were  suppliers  to  the  microelectronics  industry, 
particularly  those  that  developed  and  manufactured  the  equipment  used  to  produce 
microchips.  This  program  enjoyed  bipartisan  support,  primarily  because  it  was  coupled 
to  a  government  mission — that  of  national  security.  Even  so,  it  was  widely 
acknowledged  that  SEMATECH  addressed  larger  issues  than  national  security  and, 
issues  concerning  the  nation’s  future  economic  strengths  in  areas  which  are  critical  to 
information  technology. 

VHSIC.  In  the  early  1980s,  DoD  invested  approximately  one-half  billion  dollars 
in  the  Very  High  Speed  Integrated  Circuit  (VHSIC)  program.  Much  of  that  money  went 
to  the  aerospace  industry,  for  whom  microelectronics  was  not  their  core  business.^  As  a 


^  Review  Note:  VHSIC  experiences  point  out  the  pitfalls  of  trying  to  develop  cutting-edge  technologies 
in  a  relatively  isolated  military  industrial  sector  with  insufficient  links  to  and  competition  with 
commercial  firms. 

®  Review  Note:  Is  it  true  that  only  aerospace  firms  benefited  from  VHSIC,  or  were  some  of  the  firms 
really  military  contractors  such  as  Harris  Corporation,  and  defense  divisions  of  large  firms  such  as 
Texas  Instruments  and  IBM? 
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result  of  its  structure,  this  program  generally  failed  to  channel  government  funding  to  the 
key  suppliers  of  microelectronic  circuits.  Although  the  program  achieved  some  very 
important  advances  in  the  design  of  microcircuits,  and  many  of  the  microcircuit  design 
tools  that  exist  today  can  be  traced  back  to  these  investments,  the  program  nonetheless 
failed  to  meet  its  objective  of  improving  the  capabilities  of  the  microelectronics  supplier 
base. 

Flat  Panel  Displays.  The  recent  decision  to  proceed  with  government  support  for 
a  flat  panel  industry  has  been  controversial.  The  United  States  lags  behind  Japan  in  the 
development  of  this  technology  that  emerged  from  a  joint  investment  between  Japanese 
government  and  industry.  In  the  last  few  years  flat  panel  displays  have  been  recognized 
as  a  critical  technology  for  national  security.  Proponents  say  there  is  a  compelling  need 
for  the  military  to  have  access  to  both  the  on-shore  design  and  manufacture  of  displays; 
that  Japan  enjoys  the  advantage  of  early  development  for  which  the  government  can 
compensate;  and  that  the  flat  panel  design  and  manufacturing  are  of  fundamental 
importance  for  the  development  of  other  microelectronic  capabilities.  Others  have 
suggested  that  the  first  mover  may  be  at  a  disadvantage  because  the  technology  is 
evolving  in  new  directions  beyond  the  particular  one  being  produced  today The  United 
States  flat  panel  industry  is  best  described  as  weak,  while  off-shore  competitors  are  very 
strong.  It  remains  at  issue  whether  or  not  the  advantage  of  foreign  competitors  can  be 
overcome  through  government  and  private  investments. 

Composite  Materials.  Composite  materials  are  of  vital  importance  to  the  military. 
The  fate  of  the  composites  industry,  however,  has  been  historically  tied  to  large  military 
procurements  such  as  the  B-2  bomber.  When  such  programs  were  terminated  or  curtailed 
after  the  end  of  the  Cold  War,  the  market  demand  for  composites  dropped  by  more  than  a 
factor  of  four,  leaving  the  industry  scrambling.  In  addition,  the  commercial  sector  for 
composite  materials,  particularly  the  high  end  of  the  materials,  was  being  driven  offshore 
by  the  sporting  goods  market,  which  the  Asian  countries  captured  in  the  1980s.*®  **  It 


®  Review  Note:  Is  this  really  a  practical  possibility?  Japan’s  investment  in  flat  panel  displays  is  huge. 
Where  would  a  comparable  investment  in  the  U.S.  come  from?  Perhaps  a  more  persuasive  argument  is 
that  Korea  and  Taiwan  could  work  together  in  the  business  and  undermine  Japan’s  (primarily  Sharp’s) 
market  power. 

*®  Sporting  goods  constitute  a  very  profitable  segment  of  the  industry.  A  $200  tennis  racket  contains 
only  about  $15  of  composites,  yet  the  racket  is  marketed  based  on  the  composites. 

**  Review  Note:  Is  it  not  true  that  Japan’s  production  of  composites  for  commercial  aircraft  (e.g.,  the 
Boeing  777)  is  rather  important  to  their  market  share? 
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was  clear  that  defense  requirements  could  no  longer  sustain  the  industry  in  a  way  that 
would  satisfy  critical  defense  needs.  This  situation  led  to  a  search  for  alternate  markets 
for  composites,  both  in  and  out  of  the  aerospace  business.  Under  the  Technology 
Reinvestment  Project  (TRP)  a  program  was  therefore  established  at  Pratt  and  Whitney  to 
rapidly  insert  composites  into  turbine  engines.  As  a  result,  new  sources  of  demand  for 
composites  relevant  to  defense  are  coming  on-line.  TRP  has  also  supported  investments 
in  construction  which  may  be  applied  both  to  military  purposes  and  to  earthquake 
proofing  of  bridges  in  California. 

Commercial  Sector  R&D  Investments 

In  the  commercial  microelectronics  and  computer  industries  there  is  no  longer  any 
real  vertical  integration. Each  firm  makes  investments  in  those  areas  where  it  has  a 
particular  expertise  or  “core  competency.”  Where  a  firm  does  not  have  in-house 
expertise  it  instead  depends  on  other  companies.  For  example,  a  firm  may  have  expertise 
in  graphics,  microprocessors,  digital  media,  parallel  processing  systems,  and  user 
interfaces.  But  it  might  lack  expertise  in  complementary  technologies  such  as 
semiconductor  processing,  display  and  storage,  applications  programming  networks, 
consumer  electronics  content,  or  media  distribution.  To  complete  the  set  of  technologies 
necessary  to  compete  and  do  business,  it  becomes  necessary  to  form  partnerships  with  a 
large  number  of  other  companies  from  a  variety  of  technological  areas  become  necessary. 

The  investment  environment  in  microelectronics  is  therefore  extremely 
challenging.  Even  though  it  is  a  very  vibrant  industry,  this  is  also  extremely  competitive, 
and  that  competitive  environment  puts  constant  downward  pressure  on  margins.  As  a 
consequence,  over  the  last  10  years  there  has  been  a  reduction  in  the  percentage  of 
revenue  devoted  to  R&D,  and  this  appears  to  be  a  trend. 

Short  product  cycles  are  another  source  of  pressure.  In  the  microelectronics 
industry,  2  years  is  about  as  long  as  a  product  cycle  can  last  if  a  firm  is  to  have  continued 
success  in  the  meu:ketplace.  And  the  turnover  in  products  is  staggering — typically,  over 
50  percent  of  a  firm’s  revenues  come  from  products  that  take  less  than  eighteen  months 
to  develop.  Old  products  simply  become  obsolescent  and  unmarketable. 


Review  Note:  There  still  is  vertical  integration  in  such  markets  in  Japan.  Also,  firms  in  the  United 
States  such  as  IBM  are  certainly  to  be  regarded  as  verticeilly  integrated. 
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Most  firms  in  the  microelectronics  industry  are  placing  less  emphasis  on  basic  and 
applied  research  and  more  emphasis  on  the  development  of  products  and  processes  for 
the  marketplace.  Even  in  areas  of  core  expertise  firms  tend  to  rely  on  other  parties, 
primarily  universities,  for  longer  range  research.  This  does  not  mean  an  absence  of  all 
long-range  activities,  for  example  such  undertakings  as  technical  and  research 
conferences  for  company  developers  and  researchers.  Because  the  industry  is  fueled  by  a 
constant  stream  of  Ph.D.s  from  university  research  projects,  there  is  a  need  to  publish  at 
these  conferences  to  attract  new  talent.  In  fact,  some  firms  explain  some  of  their  most 
important  secrets  at  public  forums  just  to  attract  bright  new  employees — a  critical 
requirement  if  a  company  is  to  continue  to  prosper. 

Therefore,  in  the  commercial  electronics  and  computer  industries  most  private 
sector  technology  investments  are  short  term,  and  the  sizes  of  these  investments  are 
shrinking.  So  far,  university  partners  have  enabled  firms  to  find  and  take  advantage  of 
some  breakthroughs  in  fundamental  science  and  technology,  primarily  at  the  systems 

level.  13 

Advances  in  component  technology  are  continuing — at  least,  today — at  a  frantic 
pace,  primarily  among  suppliers.  Many  of  these  suppliers  are  often  very  small 
companies,  but  because  the  pace  of  development  in  this  area  is  part  of  what  drives  short 
product  cycles,  successful  firms  in  many  cases  depend  on  very  small  companies  for 
critical  new  component  technologies.  In  essence,  suppliers  represent  a  form  of  R&D 
investment,  although  to  an  outsider  it  may  appear  as  vendor  relations,  not  R&D 
investment.  I'l 

In  return,  willingness  to  become  an  important  supplier  for  large  companies  has  so 
far  enabled  many  small  and  start-up  computer  and  electronics  firms  to  become  some  of 
the  industry’s  most  innovative  firms.  Stable  customers  on  the  part  of  large  firms 
therefore  allow  small  firms  to  obtain  the  necessary  cash  flows  to  make  investments  they 
need  to  fulfill  industry  expectations  for  new  products,  processes,  and  technologies. 

In  the  commercial  microelectronics  industry,  partnering  with  other  firms  to  spread 
the  risk  of  investments  is  also  common.  For  instance,  Silicon  Graphics  has  a  partnership 


^3  Review  Note:  It  is  important  to  note  that  much  of  the  money  for  university  research  comes  from  the 
government,  so  that  companies  are  relying  on  the  government  for  a  considerable  portion  of  the  basic 
research  they  consume. 

Review  Note:  Important  point. 
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with  Time  Warner  to  develop  interactive  TV,  a  partnership  with  Nintendo  for  graphics 
technology  and  video  games,  and  a  partnership  with  AT&T  for  video  on  demand  servers 
and  network  technology.  These  are  big  projects  that  are  strategic  gambles — attempts  to 
open  new  areas  and  new  markets. 

Defense  Sector  R&D  Investments^^ 

Defense  companies,  particularly  those  attempting  to  shift  from  defense  to 
commercial  markets  in  the  face  of  DoD  downsizing,  encounter  a  vast  difference  in 
commercial  firms’  culture  and  approach  to  business.  Defense  firms  understand  the 
defense  market  from  the  perspective  of  their  long-term  customers.  They  know  how 
defense  markets  act  and  react  to  changing  environments.  On  the  other  hand,  defense 
firms’  executives  freely  admit  that  they  have  limited  understanding  of  commercial 
markets  and  innovation  processes,  because  they  lack  practical  experience. 

Nevertheless,  defense  firms  are  attempting  to  enter  the  commercial  marketplace, 
albeit  slowly.  The  pace  of  transformation  from  defense  to  commercial  markets  is 
evidenced  by  the  level  of  R&D  spending  by  large  firms,  where  the  majority  of  invest¬ 
ments  still  target  the  development  of  systems  for  DoD.  On  the  other  hand,  as  defense 
budgets  go  down  it  becomes  harder  and  harder  for  many  defense  firms  to  justify 
continued  expenditures  in  shrinking  markets.  In  particular,  the  metrics  for  determining 
the  allocation  of  new  investments  among  defense  and  commercial  opportunities  are 
difficult  to  derive  and  not  clearly  interpretable.  For  instance,  should  internal  budgets  be 
proportional  to  the  annual  defense  budgets,  taking  proportionate  reductions  every  year? 
Or,  should  investment  follow  a  firm’s  5-year  projection  of  bookings?  While  the  projec¬ 
tion  of  bookings  may  be  the  best  criterion  from  a  theoretical  point  of  view,  unfortunately 
there  is  widely  held  belief  in  industry  that  long-term  projections  are  not  reliable. 

Inevitably,  defense  firms  behave  like  commercial  firms  when  it  comes  to  their 
customers.  There  is  strong  coordination  with  military  customers  which  is  built  upon 
years  of  experience  with  military  procurement  practices  and  requirements,  Commercial 


Review  Note :  One  reviewer  observed  that  the  paper  is  too  strongly  oriented  towards  defense  and  that 
this  section  should  be  completely  omitted.  The  rationale  is  that  inclusion  of  the  apparent  defense  bias 
implies  that  a  dual-use  strategy  depends  on  defense  firms  entering  into  commercial  pursuits.  This  is 
not  a  highly  likely  outcome;  in  fact,  defense  firms  seem  to  be  surviving  by  combining  through  mergers 
and  acquisitions. 

Review  Note:  It  is  more  likely  that  dual-use  will  succeed  when  the  military  begins  to  procure  more 
from  the  dynamically  competitive  commercial  sector. 
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investments  by  defense  firms,  on  the  other  hand,  are  largely  experimental  and  “buffered” 
against  failure  by  playing  against  successful  defense  technologies.  This  is  a  dual-use 
strategy  that  attempts  to  leverage  defense  expertise  which  may  have  a  high  commercial 
payoff. 

Another  strategy  in  the  defense  business  is  to  compete  with  other  defense  firms  on 
the  basis  of  weapon  system  affordability  and  capabilities.  This  is  done  by  taking 
hardware/software  systems  already  supplied  to  the  military  and  making  them  more 
affordable — similar  to  competition  in  commercial  markets  among  innovating  firms 
competing  on  the  basis  of  price  for  market  share.  Examples  of  such  commercial 
endeavors  in  the  dual-use  of  technologies  are  found  in  the  telecommunications  and 
medical  systems  areas,  in  transportation  and  movement  of  goods  across  the  world,  and  in 
imaging  systems. 

Insofar  as  a  defense  firm  is  willing  to  take  the  plunge  into  commercial  markets, 
such  decisions  are  made  on  the  basis  of  anticipated  profitability.  In  many  cases 
technology  investment  decision  making  relies  99  percent  on  this  “simple”  criterion.  This 
reflects  the  emphasis  on  short  turn  returns  on  investments  held  by  industry  executives 
with  a  primarily  financial  view  of  the  world. 

The  drive  for  short-term  returns  in  the  private  sector  results  in  a  focus  on 
investments  that  can  yield  immediate  returns.  An  important  role  for  the  government  is  to 
enable  the  private  sector  to  take  a  longer  term  view  by  defraying  some  of  the  risk 
associated  with  extended  time  horizons.  Such  government  efforts  should  stimulate,  not 
simply  supplement,  private  sector  investment.  This  will  make  it  easier  for  executives  in 
industry  to  justify  making  their  own  investments  where  there  is  a  potential  for  longer 
term  payoffs.*”^ 

The  Role  of  Peer  Review 

The  peer  review  issue  is  not  whether  we  should  give  money  to  science  and 
technology.  It  is  concerned  with  the  distribution  of  resources  to  particular  scientists  and 
engineers  or  particular  projects.  How  do  we  evaluate  and  judge  what  our  science  and 
technology  projects  should  be?  How  do  we  evaluate  their  performance?  We  have  to 
make  judgments  about  the  returns  we  expect,  and  this  is  extremely  difficult  ex  ante. 


Review  Note:  This  paragraph  implies  that  government  dual-use  policy  is  oriented  towards  bringing 
defense  firms  into  commercial  markets,  which  is  not  the  case. 
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Role  of  Intuition.  Despite  all  the  attempts  at  quantification  in  science  and 
technology,  we  have  historically  used  qualitative  indicators  to  make  most  funding 
decisions.  That  is,  we  become  imprecise  rather  then  precise,  so  all  the  positive  virtues 
that  we  normally  think  of  as  scientific  become  hard  to  recognize.  Of  course,  anyone  who 
has  studied  the  nature  of  the  scientific  enterprise  knows  how  important  intuition  is  to  the 
pursuit  of  scientific  objectives.  So,  in  fact,  the  image  of  scientific  precision  and  rigor  is  a 
bit  overdrawn.  Many  of  the  greatest  leaps  of  science  have  been  leaps  of  intuition  that 
have  not  been  based  on  line-by-line-deductive  reasoning.  Such  outcomes  are  impossible 
to  predict.  They  remain  to  be  explained  retrospectively. 

Peer  review,  which  must  account  for  intuition  as  well  as  prediction,  is  an 
imperfect  and  often  qualitative  approach  to  making  judgments  about  the  likely  outcome 
of  investments  in  scientific  research,  but  it  is  the  best  available  method  for  the  ex  ante 
allocation  of  resources.  If  we  are  aware  of  the  biases  inherent  in  a  peer  review  process, 
we  can  try  to  compensate  for  them.  Those  who  do  peer  review  well  are  aware  of  bias 
problems  and  attempt  to  correct  for  them.i^ 

Bias  Effects.  Two  primary  types  of  bias  occur  in  peer  review.  The  first  type  is 
the  fad  effect.  There  are  fads  in  all  disciplines.  These  are  certain  subjects  that  become 
very  popular  and  generate  sizable  literature.  When  a  peer  community  is  asked  about  the 
scientific  interest  of  a  proposed  project,  there  is  a  bias  that  what  is  interesting  is  what  a 
large  number  of  investigators  are  pursuing.  If  there  are  many  investigators  pursuing 
similar  research,  then  there  are  many  voices  to  say,  “This  is  really  interesting,”  without 
much  consideration  of  the  potential  social  return  from  the  particular  enterprise.*^ 

Without  a  way  to  relate  scientific  and  technological  endeavors  to  prospective 
social  returns,  peer  review  entails  a  certain  degree  of  myopia.  For  instance,  in  the  1970s 
it  was  very  hard  to  obtain  research  support  for  investigations  of  technology  and 
technological  change  from  the  NSF  economics  program.  Since  no  one  was  doing  such 
research,  it  was  presumed  not  worthy  of  investigation.  Similarly,  we  recognize  today 
how  important  environmental  externalities  are  to  determining  the  quality  of  life.  Yet  20 


Review  Note:  Important  point. 

Review  Note:  Are  social  returns  a  proper  consideration  for  scientists  and  technologists? 
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years  ago  this  was  not  a  subject  in  which  there  was  much  economic  research,  and  it  was 
therefore  hard  to  get  projects  approved  through  the  peer  review  system.20 

A  second  related  bias  that  occurs  in  peer  review  is  the  fraternity  effect.  In  this 
case  investigators  act  to  positively  reinforce  each  other’s  work  or  interests.  In  the  field  of 
economics,  it  is  interesting  to  note,  there  are  strong  fraternity  effects  in  some 
subspecialties  and  strong  property  rights  effects  in  others.  Property  rights  effects  occur 
when  peer  reviewers  working  in  an  area  respond  to  a  new  proposal  by  indicating  that  the 
work  is  already  being  solved  by  someone  else.  Those  in  the  fraternity  work  to  prevent 
others  from  entering  and  disturbing  their  investigative  turf.  The  sociologies  of  subgroups 
within  a  field  differ  markedly,  and  can  bias  the  outcome  of  peer  reviews  in  ways  that  are 
obscure  to  the  outsider. 

Good  peer  review,  therefore,  takes  known  biases  into  consideration  and  tries  to 
protect  against  them.  For  instance,  program  officers  in  economics  have  long  had  open 
discussions  of  bias  problems  and  have  studied  various  subgroups  within  the  discipline. 
When  the  peer  reviews  are  received,  the  program  officers  take  these  biases  into  account 
and  try  to  offset  them. 

Ex  post  v^.  ex  ante  evaluation.  It  is  important  to  recognize  that  project  evaluation 
is  nothing  more  than  cost  benefit  analysis  applied  to  ex  post  assessment.  We  do  a  lot  of 
cost-benefit  analysis,  and  it  is  all  ex  ante.  Rarely  do  we  return  after  a  project  is  complete 
and  ask  if  estimates  about  the  costs  and  the  benefits  were  any  way  commensurate  with 
what  actually  resulted.  Ex  post  assessments  have  the  function  of  going  back  and 
reviewing  the  returns,  the  costs,  and  the  benefits  of  a  project.  This  is  not  only  to  satisfy 
intellectual  curiosity.  It  is  important  to  our  understanding  of  the  role  of  assessment.^!  22 

Ex  post  assessments  provide  a  much  better  view  about  how  well  we  are  doing  in 
resource  allocation,  designing  incentives,  and  ex  justification.  In  particular,  we  must 
become  aware  of  areas  where  there  may  be  systematic  deviations  of  our  expectations 
from  realization.  If  we  can  identify  systematic  errors  in  judgment,  then  we  can  attempt  to 
undo  the  systematic  biases. 23 


^  Review  Note;  This  is  not  altogether  true.  In  1975  both  the  Council  on  Environmental  Quality  and 
Resources  For  the  Future  were  very  interested  in  this  subject. 

2!  Review  Note:  Important  point. 

22  Review  Note:  As  written  this  paragraph  is  inflammatory  for  the  scientific  community. 

23  Review  Note:  Good  statement  of  the  issue. 
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For  instance,  NSF  has,  over  the  years,  tried  to  learn  whether  there  is  a  bias  against 
young  scholars  or  old  scholars.  There  have  been  allegations  in  the  past  that  young 
scholars  have  trouble  breaking  into  the  group  of  sponsored  researchers.  That  is  a 
question  that  is  amenable  to  statistical  analysis.  You  can  determine  the  rate  of  entry  of 
young  people,  determine  when  do  they  come  in,  and  what  their  records  are  at  that  time. 
As  it  turns  out,  the  evidence  is  that  there  may  be  discrimination  against  older  scholars, 
rather  than  younger  ones.  NSF  may  have  become  so  concerned  about  younger  scholars 
that  peer  review  now  is  biased  against  older  scholars.  Furthermore,  the  people  who  have 
the  time  to  sit  on  the  peer  review  panels  tend  to  be  younger  people,^^  and  they  tend  to 
take  the  view:  “What  theorems  have  you  proven  in  the  last  three  hours?”  If  the  answer 
to  that  is,  “under  three,”  the  view  is  that  you,  obviously,  have  been  losing  your  marbles 
and  no  longer  warrant  continued  support. 

One  must  also  try  to  be  careful  about  when  qualitative  judgments  are  likely  to  be 
superior  to  quantitative  data,  and  when  they  are  likely  to  be  inferior.  If  one  sits  on  an 
admissions  committee,  such  as  those  found  in  universities,  there  is  a  widespread  belief 
among  the  members  that  they  have  an  ability  to  bring  to  bear  qualitative  judgments  on 
what  is  likely  to  make  a  successful  student — all  of  us  have  an  enormous  amount  of 
credence  in  our  own  abilities  in  this  respect.  But  that  is  something  that  is  amenable  to 
statistical  verification.  The  few  studies  that  have  been  done  seem  to  suggest  that 
admissions  committees  have  absolutely  no  ability  to  make  judgments.  That  is  to  say  if 
you  look  at  performance  based  on  what  would  have  happened  if  you  used  a  regression 
model,  you  do  better  than  with  an  admissions  committee.  An  admissions  conunittee’s 
judgments  add  noise  to  the  selection  process.  This  stands  as  a  caution  in  all  of  our 
confidence  about  the  value  of  qualitative  information  and  our  strong  ability  to  make  good 
and  sound  judgments. 


Review  Note:  Is  this  really  true?  Are  review  panels  usually  composed  of  younger  people?  Or  is  this 
a  function  of  the  particular  discipline? 
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Qualitative  Assessments  of  Research  Results^s 

Before  a  project  is  started,  there  are  assumptions  that  underlie  its  expectations. 
These  include  assumptions  about  how  the  market  is  going  to  develop,  and  assumptions 
about  how  profitable  the  business  is  going  to  be.  But  ex  post  evaluation  occurs  after  the 
initial  investment  when  decision  makers  look  back  and  ask,  “Did  our  investment — which 
the  corporation  paid  for  as  part  of  its  total  bottom  line  and  profitability — put  us  into  a 
reasonable  position?”^^ 

The  benefits  most  often  looked  for  and  talked  about  are  major  breakthroughs  that 
will  lead  to  major  products  and  put  a  firm  ahead  of  the  competition.  But  those  may  not 
be  the  most  important  benefits.  A  firm  needs  the  ability  to  catch  up  with  companies  or 
institutions  that  make  the  breakthrough  first.  No  matter  how  a  firm  invests  in  research,  it 
is  not  going  to  make  all  of  the  breakthroughs.  This  means  a  firm  needs  the  ability  to 
assess  technology  from  wherever  it  comes,  and  judge  what  will  work.  It  is  equally 
valuable  to  know  what  will  not  work,  to  avoid  putting  development  dollars  into  dead 
ends.  A  firm  needs  to  be  a  wise  procurer  of  technology,  and  a  research  team  can  help, 
even  if  it  is  not  its  own  technology  supplier.  A  research  team  can  serve  as  internal 
consultants  to  assist  in  sorting  through  university  research. 

Industry’s  technology  investments  are  intended  to  develop  products  or  services  on 
which  the  company  can  make  a  profit,  and  this  profit  can  be  measured.  Profitability 
depends  upon  the  response  of  customers  and  the  response  of  competition.  It  is  very 
important  to  be  able  to  assess  revenue  streams  to  decide  where  to  make  technology 
investments.  A  company’s  report  card  shows  up  every  day  in  The  Wall  Street  Journal. 
R&D  investments  are  a  part  of  the  equation.  The  question  is  this:  Of  current  revenues, 
how  much  should  be  returned  to  shareholders  and  how  much  should  be  put  into  R&D, 
and  where?  How  much  money  really  goes  into  technology  versus  development  versus 
engineering  and  product  versus  process  versus  infrastructure? 


^  Review  Note:  This  whole  section  refers  to  ex  post  evaluation  of  private  R&D.  The  Round  Table  was 
about  metrics  for  public  and  cooperative  investments.  This  gives  the  impression  that  public 
investments  can  be  evaluated  in  the  same  way  as  private  ones.  Also,  the  NIST  approach  to  ex  post 
evaluation  is  completely  left  out  here,  even  though  it  is  the  best  we  have  in  the  government  for  public 
sector  technology  investments. 

^  Review  Note :  There  is  a  need  to  make  the  distinction  between  science  and  technology  here.  The 
Round  Table  was  concerned  with  technology.  Ex  post  assessments  are  already  difficult  enough  in  the 
technology  area,  and  are  even  more  difficult  (if  not  impossible)  in  the  area  of  science  (basic  research). 
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Ex  post  methods  of  evaluation  are  really  a  continuation  of  the  ex  ante  methods. 
These  ex  ante  methods  are  frequently  referred  to  as  a  business  case — a  detailed  analysis 
of  the  various  expenses,  including  capital  expenses,  revenues,  and  the  eventual  profits.  In 
any  well-run  organization,  the  business  case  for  the  whole  project  has  to  be  undertaken 
before  the  project  starts.  It  should  be  updated  regularly,  and  there  should  also  be  an 
analysis  when  the  project  is  complete. 

When  is  a  project  complete?  Some  would  say:  When  the  design  is  turned  over  to 
the  manufacturer — an  answer  that  is  obviously  incorrect.  Some  would  say:  It  is  when 
the  first  product  is  sold  to  a  customer — a  response  that  might  be  correct.  Some 
companies  say:  It  is  when  you  first  break  even  on  the  total  project. 

For  ongoing  assessments,  the  judgments  of  peers,  mostly  within  the  company,  can 
be  used.  It  is  also  possible  to  solicit  the  judgments  of  peers  outside  the  firm  as  to  whether 
the  quality  of  the  work  you  are  doing  is  good.  Firms  should  also  review  what  were  their 
starting  points  and  ask  themselves:  “Are  they  still  achieving  the  attractive  market  results 
that  they  thought  might  be  there?”  “Is  tactical  performance  up  to  snuff?”  “Is  it  going  to 
allow  the  firm  to  achieve  the  market  share  that  they  thought  they  were  going  to  get?” 
“Are  they  on  schedule?”  “Is  the  project  cost  going  to  overrun?”  One  should  therefore 
look  continuously  at  the  relevance  of  what  one  is  doing  in  the  business  because  this  will 
change  as  business  and  technology  prospects  change.  As  part  of  this  ongoing  process  of 
evaluation,  one  therefore  heavily  depends  upon  the  informed  judgment  of  “smart”  people. 
As  part  of  these  continual  assessments  one  must  also  face  the  issue  of  whether  or  not  it 
would  be  prudent  to  terminate  a  program  rather  than  continue  it. 

Finally,  the  degree  of  interaction  between  the  firm’s  research  staff  and  its 
development  staff  should  be  continually  reviewed.  Lack  of  effective  communication  here 
suggests  that  the  project  may  not  be  on  track.  It  also  means  that  the  chance  of  taking 
research  results  and  turning  them  into  useful  products  or  services  is  greatly  diminished. 

Econometric  Assessments 

Econometric  measures  of  the  results  of  technology  investments  are  quantitative, 
but  abstract.  Econometric  analysis  is  based  on  theories  of  economic  growth  which 
originated  in  the  1950s,  when  Robert  Solow  and  Moses  Abramovitz,  among  others,  tried 
to  evaluate  the  determinants  of  the  rate  of  productivity  growth  in  the  U.S.  and  other  major 
economies.  They  discovered  a  mysterious  residual — a  large  fraction  of  growth  which 
appeared  to  be  unexplained  by  capital  investment,  labor,  or  other  obvious  sources. 
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Economists  searched  for  explanations  of  this  residual,  considering  such  factors  as 
education.  Solow  suspected  that  the  primary  determinant  of  the  residual  was  innovation, 
and  attributed  its  cause  to  “technology”  or  “technological  change.”  Economists  consider 
R&D  investment  to  be  a  source  of  this  technology  residual. 

It  also  appears  that  the  technology  residual  is  associated  with  “spill-overs”  to 
firms  other  than  the  initial  innovator.  The  economics  community  strongly  holds  that 
R&D  investments  not  only  benefit  the  innovator,  but  also  lead  to  the  creation  of  so-called 
public  goods.  The  innovations  of  one  firm  lead  to  indirect  benefits  for  other  firms. 

Economists  also  hold  strongly  that  traditional  market  incentives  will  not 
necessarily  yield  a  rate  of  return  to  innovators  which  is  as  great  as  the  one  for  society  at 
large.  R&D  tax  credits,  for  example,  are  justified  by  the  Council  of  Economic  Advisors 
as  a  means  of  increasing  the  rate  of  return  to  entice  firms  to  make  investments  which  can 
have  broad  social  benefits. 

Econometric  analysis,  as  performed  by  economists  such  as  Zvi  Griliches  at 
Harvard,  includes  a  search  for  the  spill-over  effects  which  can  help  explain  overall 
productivity  growth,  including  the  growth  residual.  Analysts  seek  to  understand,  for 
example,  how  spill-overs  may  be  employed  as  important  policy  tools.  Econometrics 
seeks  to  reveal  how  activity  in  one  industry  or  firm  affects  other  firms  and  other 
industries. 

One  of  the  strengths  of  the  econometric  approach  is  that  one  can,  in  principle, 
avoid  the  selection  bias  that  creeps  in  when  performing  case  studies.  For  instance,  Edwin 
Mansfield  did  a  series  of  case  studies  which  looked  at  innovation  within  companies, 
focusing  on  investment  costs,  returns,  and  the  extent  to  which  returns  spilled  over  into 
other  companies.  But  there  is  always  the  possibility  that  such  a  methodology  will  be 
biased  toward  winners,  because  one  is  studying  companies  that  are  known  as  innovators. 
There  is  a  danger  in  the  case  study  approach  that  one  will  select  ex  post  those  companies 
or  research  projects  with  a  higher  than  average  return.  This  leads  to  an  overstatement  of 
the  overall  rate  of  return.  Econometric  analyses  which  look  at  the  performance  of  an 
entire  industry,  or  a  whole  set  of  industries,  can  be  more  inclusive,  by  including  R&D 
investments  that  may  not  have  paid  off  as  well  as  those  that  did. 

A  second  strength  of  the  econometric  approach  is  that  it  can  account  for  spill¬ 
overs  that  are  not  apparent  to  the  case  study  worker.  Mansfield  did  a  superb  job,  but  he 
was  limited  by  his  ability  to  see  where  technology  development  in  one  firm  affected 
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operations  in  other  firms.  In  recent  econometric  literature,  there  have  been  attempts  to 
track  spill-overs  more  widely.  People  have  looked  at  technology  relationships  between 
supplier  industries  and  downstream  industries.  For  example,  they  have  examined  patent 
citations  to  trace  technological  links  among  industries  and  then  estimated  the  these  effects 
through  statistical  methods.  Ideally,  one  would  like  to  look  at  links  worldwide,  since  the 
technology  that  is  developed  in  a  firm  in  the  United  States  might  be  the  result  of  work 
that  was  done  in  Japan  or  Germany,  and  vice-versa.  Unfortunately,  such  research  has  not 
been  done,  although  considerable  work  in  the  United  States  has  tried  to  take  a  broader 
view  of  spill-overs. 

What  are  some  of  the  weaknesses  of  the  econometric  approach?  There  is  always 
the  danger  that  one  will  arrive  at  a  correlation  and  not  know  if  it  is  meaningful, 
particularly  with  respect  to  the  direction  of  causality.  This  is  particularly  difficult  with 
R&D  because  there  is  no  quick  response  to  changes  in  R&D  budgets  in  either  the  output 
or  the  productivity  of  the  industry.  There  are  long  variable  lags  between  R&D,  its  effect, 
and  the  number  of  patents  or  productivity  changes  that  result  in  an  industry.  The 
economics  profession  has  developed  many  sophisticated  techniques  for  getting  around 
these  issues,  but  at  the  end  of  the  day  there  is  always  the  element  of  doubt. 

A  second  weakness  of  econometric  methods  is  that  they  are  not  of  great  use  to 
policy  makers  for  reviewing  a  particular  investment.  Econometrics  is  not  a  substitute  for 
the  men  and  women  making  wise  decisions  about  which  project  to  fund  because  such 
decisions  are  done  on  a  level  which  is  much  more  specific.  Rather,  econometrics  is 
helpful  in  understanding  why  it  is  important  to  support  R&D  in  general,  and  in  helping  to 
determine  if  the  overall  magnitude  of  effort  is  sufficient. 

There  are  important  technical  problems  in  making  econometric  estimates.  For 
instance,  it  is  particularly  difficult  to  measure  the  amount  of  R&D  that  is  being  done 
because  reported  R&D  data  may  be  missing  a  lot  of  the  action.  It  is  also  difficult  to 
measure  inputs  and  outputs.  As  an  example,  until  some  years  ago  the  output  of  the 
computer  industry  was  imperfectly  measured.  It  was  assumed  that  the  price  of  computers 
remained  constant  over  time.  So  measuring  productivity  in  the  computer  industry  was, 
obviously,  a  flawed  exercise.  The  Department  of  Commerce  then  went  into  a  major 
project  to  construct  new  computer  price  indexes,  using  “hedonics”;  that  is,  trying  to 
assess  the  characteristics  of  the  computer,  the  number  of  MIPS,  the  size  of  the  hard  disk 
and  so  on.  As  a  result,  statistics  reveal  considerably  more  output  and  productivity  growth 
generated  in  the  computer  industry  than  before.  In  a  lot  of  industries  price  measures  are 
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still  poor,  and  it  is  hard  to  capture  the  return  to  R&D  investments.  This  is  a  fundamental 
problem  in  assessing  overall  productivity  and  performance  of  the  economy,  but  it  is  of 
particular  concern  for  R&D  assessment.^^ 

Case  Studies 

Case  studies  may  be  used  to  help  decision  makers  make  better  judgments  than 
they  might  make  based  on  abstract  data,  and  they  provide  clear  evidence  for  the  value  of 
the  science  and  technology  enterprise.  This  point  is  important  because  of  recent 
skepticism  about  the  utility  of  certain  kinds  of  government  technology  programs.  The 
accusation  is  not  just  that  the  government  “builds  bad  bridges,”  but,  rather,  that 
government  should  not  be  building  bridges  at  all.  The  question  may  be  asked,  “Why 
shouldn’t  government  just  spend  more  money  on  tax  incentives,  reduce  its  regulatory 
role,  and  let  things  proceed  on  their  own?”  Case  studies  provide  an  explanation  about 
how  industrial  competition  operates,  and  how  companies  come  to  succeed  in  their  mar¬ 
ketplaces.  Case  studies  are  a  particularly  important  part  of  ex  post  justification.  They 
provide  nontechnical  people  the  ability  to  assess  why  that  it  is  in  the  national  interest  for 
more  beautiful  theorems  to  be  proven,  and  more  technology  investments  to  be  made. 

Case  studies  also  help  reveal  the  role  of  government  scienee  and  technology  in 
global  competition.  The  pace  of  technology  has  sped  up.  Most  companies  are  no  longer 
vertically  integrated  in  the  way  IBM  traditionally  was,^*  and  this  changes  the  decision  of 
what  is  to  be  developed  in-house  and  what  is  to  be  obtained  from  the  global  pool  of 
technology.  Companies  are  increasingly  dependent  for  R&D  on  other  firms  and 
universities.  This  suggests  that  one  role  for  government  is  to  make  certain  that  public 
programs  contribute  to  a  portfolio  of  technology  that  can  help  companies  compete  in 
global  markets. 

We  therefore  speak  of  a  “supply  base,”  which  is  the  set  of  technological 
components,  subsystems  and  systems  that  are  available  to  companies  and  enable  them  to 
produce  today  and  innovate  for  tomorrow.  So  one  task  for  case  studies  is  to  investigate 
how  government  projects  contribute  broadly  to  a  supply  base. 

Case  studies  also  support  the  derivation  of  general  principles  of  technology 
investment  by  comparison  across  cases.  It  is  not  just  whether  a  particular  case  led  to 


^  Review  Note:  Good  section! 

Review  Note:  IBM  is  still  one  company,  although  more  decentralized  than  in  the  past. 
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more  jobs,  more  patents,  etc.,  but  a  desire  to  derive  a  general  set  of  principles  that  is 
involved.  This  raises  the  question  of  the  unit  of  analysis.  Do  you  use  a  set  of  innovations 
or  an  industry?  A  series  of  case  studies  which  centered  on  the  semiconductor  industry, 
for  example,  led  its  analysts  to  emphasize  the  role  of  closed  markets,  of  constraints  on 
manufacturing  capacity,  and  of  the  ability  to  continue  to  produce  particular  kinds  of 
goods,  on  the  industry’s  ability  to  sustain  technology  development. 

Case  studies  also  reveal  ways  in  which  the  approaches  to  technology  development 
in  government  and  industry  are  incompatible.  Government  procurement  processes 
typically  take  18  months  to  award  an  R&D  study  that  may  comprise  only  nine  months  of 
actual  research  or  development,  resulting  in  a  technology  that  is  obsolete  in  12  months. 
Some  of  the  important  information  that  case  studies  provide  includes  an  analysis  of  such 
barriers  to  more  efficient  government  investments,  and  how  to  remove  them. 

The  quality  of  case  studies  may  be  uneven.  Many  are  simply  non-representative 
vignettes.  Nevertheless  there  is  a  lot  of  good  work  that  can  be  done  in  the  way  of 
detailed  multiple  studies  and  systemic  comparisons.  An  area  ripe  for  this  type  of  work 
would  be  comparisons  between  non-government  and  government  approaches  to 
technology  investment  with  the  goal  of  making  the  government  approach  more 
compatible  with  the  civilian  approach  so  that  it  can  stimulate  more  rapid  technological 
evolution. 

To  do  good  case  studies  requires  that  evaluation  measures  be  clearly  stated  at  the 
beginning  and  retained  throughout  the  program.  Data  collection  for  such  case  studies 
must  be  set  up  at  the  beginning  and  continued  throughout  the  program.  There  are  many 
examples  of  government  programs  where  the  objective  changes  quite  significantly 
subsequent  to  program  initiation 

In  summary,  the  advantage  of  a  case  study  methodology  is  twofold.  It  provides  a 
method  to  keep  track  of  the  successes  and  failures  of  particular  technology  investments. 
It  provides  a  way  to  capture  lessons  learned.  There  is  some  agreement  that  successful 
cases  provide  more  information  than  failures.  They  allow  us  to  emulate  and  build  upon 
successes.  There  may  be  too  many  combined  effects  to  assign  blame  in  the  case  of 
failures.  Case  studies  provide  lessons-leamed  which  can  assist  with  future  technology 
investment  decision  making  and  with  the  management  of  projects. 
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ROUND  TABLE  DISCUSSION 


Government  Role  in  Science  and  Technology 

Does  the  Clinton  administration  support  both  science  and  technology?  Several 
recent  studies  have  looked  at  the  School  of  Engineering  at  MIT,  which  gets  55  percent  of 
its  research  support  from  the  Department  of  Defense.  The  percentage  in  the  School  of 
Engineering  at  Stanford  is  about  the  same.  Yet,  the  latest  budget  that  the  administration 
sent  to  Congress  has  a  $300  million  cut  in  the  research  component  of  the  Department  of 
Defense’s  budget, 29  which  will  fall  disproportionately  on  university  engineering  research 
programs  such  as  these.  Is  the  administration  saying  by  its  actions  that  it  supports  science 
but  it  doesn’t  support  technology?^^ 

Although  DoD  support  for  university  funding  has  declined  in  absolute  terms, 
relative  to  other  cuts  in  the  defense  budget  it  has  remained  remarkably  stable.  In  a 
defense  budget  which  is  shrinking  quite  dramatically  in  real  terms  the  programs  and 
functions  that  you  might  expect  to  be  squeezed  get  squeezed.  You  can’t  squeeze  salaries 
since  they  are  fixed.  It  is  difficult  to  squeeze  people,  although  that  has  actually  been 
done  over  a  period  of  time.  So  what  is  customarily  done — and,  this  is  true  both  in  the 
private  sector  and  in  the  public  sector — is  to  squeeze  investment  accounts.  And  that,  in 
fact,  has  happened. 

Over  the  last  10  years,  defense  procurements  have  declined  by  somewhere 
between  a  third  and  40  percent  in  real  terms.  But  the  research  accounts  have  been 
protected.  That  does  not  mean  that  there  has  been  no  effect  on  science  and  technology 
from  the  dramatic  downsizing  of  the  defense  budget,  but  it  is  clear  that  the  research 
accounts  have  been  treated  better  than  any  of  the  other,  obviously  squeezable  accounts.  It 
is  extraordinary  that  the  Department  of  Defense  is  deferring  buying  weapons  systems 
which  are  manifestly  attractive  and  which  enjoy  wide  congressional  support,  and  yet  only 
took  $300  million  out  of  what  is,  after  all,  a  $2.5  billion  account,  roughly  speaking.32 


29  Fiscal  Year  not  mentioned. 

30  Review  Note:  This  is  a  nonsense  question. 

3*  Review  Note:  This  is  not  true,  people  can  be  let  go. 

32  Review  Note:  This  must  only  represent  only  the  6.1  basic  research  account.  The  overall  “tech  base” 
is  6.1  through  6.3A,  which  is  in  the  neighborhood  of  $10  to  $1 1  million. 
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As  far  as  industry  is  concerned,  there  are  shortfalls  in  the  basic  research 
necessary  to  maintain  the  generic  technology  base.  But,  the  biggest  shortfalls  are  not  in 
research,  but  development.  Comparing  the  U.S.  with  its  competitors  in  terms  of 
development  as  a  percentage  of  GDP,33  the  U.S.  has  a  $40  billion  shortfall  for  civilian 
industry  compared  with  civilian  industries  in  Japan,  Germany,  or  France.  Thus,  even 
when  bright  ideas  abound,  insufficient  resources  are  available  to  do  their  development. 
Subsequent  to  development,  there  is  a  tenfold  requirement  for  capital  and  marketing 
investments  to  bring  developed  technologies  to  the  marketplace. 

The  problem  is  not  that  industry  is  underinvesting  in  development  for  the  products 
it  is  manufacturing.  The  problem  is  a  declining  world  market  share.  Over  the  past  few 
decades  the  United  States  has  been  losing  world  market  share.  There  are  some  markets 
where  the  U.S.  is  just  not  present.  ^5  A  prime  example  is  consumer  electronics.  If  the 
United  States  were  to  manufacture  its  share  of  consumer  electronics — that  is,  the  amount 
that  it  consumes — and  if  that  market  share  brought  with  it  a  proportionate  share  of 
investment  in  development,  this  shortfall  would  disappear. 

One  of  the  things  the  United  States  must  consider  is  keeping  the  level  of  basic 
research  and  investment  in  pre-competitive  technology  properly  balanced  so  that  industry 
is  financially  able  to  take  this  technology  and  do  the  development,  manufacturing,  and 
marketing.  A  role  for  government,  if  it  wants  to  assist  high  tech  industry,  is  to  get  the 
cost  of  capital  down — ^not  the  cost  of  debt,  only,  but  the  cost  of  capital.36  To  do  this  one 
must  look  at  deficit  factors — high  hurdle  rates,  tax  laws  that  favor  consumption  over 
savings,  depreciation  rates.  Without  tackling  both  sides  of  this  equation,  the  nation  is 
putting  money  into  things  that  will  not  be  exploited  in  the  United  States.  In  part,  this 
suggests  that  a  billion  dollars  put  into  tax  relief  or  tax  credits  may  be  better  spent  than  a 
billion  dollars  that  goes  into  certain  technology  programs. 

We  are  in  the  midst  of  a  major  shift  in  this  country  in  resources  being  applied  to 
this  mid-term,  applied-technology  business.  The  visible  defense  budgets  have  not 


Review  Note:  It  is  important  to  be  careful  with  this  comparison  because  it  requires  one  to  understand 
what  is  in  the  portfolio  to  make  sense  of  the  investment  categories. 

^  Review  Note:  It  is  not  clear  where  the  number  $40  billion  comes  from  or  how  it  was  calculated. 

Review  Note:  This  is  only  true  in  some  industries.  On  the  whole  the  United  States  market  share  has 
been  relatively  steady. 

36  Review  Note:  National  Academies  studies  show  that  not  only  is  the  cost  of  capital  important,  but 
management  is  just  as  important. 
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substantially  reduced  R&D,  but  less  visible  budgets — such  as  IR&D,  which  in  the  past 
was  half  that  of  DoD  investments — are  down  sharply  because  they  are  based  on  a 
percentage  of  the  business  base.  In  the  commercial  sector  there  has  also  been  a  shift 
emphasizing  development  at  the  expense  of  research. 

How  should  government  make  investments  with  particular  companies  or  sets  of 
companies  whose  purpose  is  to  produce  spill-over  benefits?  The  fundamental  goal  for 
government  in  technology,  in  an  economic  sense,  has  always  been  infrastructure — long 
term  risky  investments.  It  has  never  been  short  term,  to  take  the  product  to  market. 

What  are  government-industry  partnerships  all  about?  The  government  brings  to 
the  table  the  ability  to  invest  for  the  longer  term  to  achieve  broader  based  economic 
benefits.  Firms  bring  to  the  table  the  ability  to  invest  in  things  for  their  own  benefit  by 
taking  particular  concepts  to  the  marketplace. 

There  are  a  lot  of  theoretical  discussions  about  whether  R&D  tax  credits  are  a 
better  way  to  solve  the  nation ’s  development  deficit  than  direct  technology  investment 
programs.  In  fact,  the  nation  has  had  R&D  tax  credits  in  place  for  many  years.  During 
that  time  there  has  been  no  concrete  evidence  that  private  R&D,  as  a  fraction  of  GDP, 
rose.  In  fact  it  may  have  been  stable  or  even  fallen  a  bit.  There  is  no  evidence  that  R&D 
tax  credits  have  done  the  kinds  of  jobs  that  Administration  technology  programs,  such  as 
ATP  and  TRP,  are  trying  to  do.  Therefore,  while  issues  concerning  tax  structures  and 
regulatory  burdens  are  important  to  industry’s  decision  making  processes,  targeted 
technology  programs  remain  absolutely  critical.  It  is  a  false  comparison  to  say  one 
approach  is  superior  to  the  other.  The  entire  system  is  in  fact  important,  and  both 
approaches  are  needed. 

Cost  share  should  be  adjusted  to  reflect  risks.  There  is  a  need  to  adjust  cost  share 
from  the  government  to  reflect  the  degree  of  risk  in  a  project.  While  the  50/50  cost  share 
mix  is  a  popular  notion,  it  is  bureaucratic  and  does  not  properly  adjust  for  risk.  There  is  a 
need  to  adjust  cost  share  to  the  risk  and  the  time  scales  involved  to  make  sure  that 
opportunities  are  available  for  universities  and  small  businesses  to  be  involved.  In  some 
cases  this  means  the  willingness  and  ability  to  accept  matching  funds  in  kind — not,  just 
direct  monetary  contributions. 

DoD  has  a  relatively  simple  technology  transitioning  problem  since  it  has  a  fairly 
captive  customer  base.  The  transition  should  be  much  less  complicated  than  it  is  in  the 
commercial  world.  Yet  DoD  has  never  done  well  even  with  its  captive  customers.  The 
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problem  stems  largely  from  the  technology  community  which,  almost  never  thinks  from 
the  customer’s  perspective.  Technologists  think  they  know  exactly  what  the  customer 
needs,  but  they  frequently  don’t  talk  adequately  to  the  customer,  There  is  a  lack  of 
communication.  If  this  is  difficult  under  the  fairly  easy  circumstances  faced  by  DoD, 
then  it  is  extra  difficult  in  the  commercial  sector.  This  deserves  a  fair  amount  of  study, 
for  instance  about  Japanese  practices.  They  seem  to  perform  much  better  in  this  respect. 

Funding  the  future  may  mean  allocating  resources  where  there  is  yet  no 
constituency.  An  essential  characteristic  of  advanced  technology  programs  is  that  their 
projects  and  ideas  create  the  next  generation’s  capabilities.  These  are  areas  for  which 
there  is  not  yet  a  constituency,  be  it  the  Defense  Department  looking  at  new  weapons 
systems,  or  be  it  future  commercial  opportunities.  This  is  especially  critical  at  the 
beginning  of  the  resource  allocation  process  where  one  is  trying  to  set  priorities.  If  one 
only  responds  to  the  loudest  voices  then  important  opportunities  may  be  missed. 

Can  review  be  used  to  assign  resources  across  disciplines?  For  instance,  how 
much  you  put  into  astronomy  versus  how  much  you  put  into  biochemistry?  It  is  difficult 
to  do  cross-discipline  comparisons,  but  there  is  still  a  role  for  peer  review,  something  that 
might  be  termed  the  “incentive  effect.”  Sometimes  the  research  community  becomes 
separated  from  the  user  community,  so  that  the  ultimate  social  return  isn’t  identified  very 
clearly  by  the  researchers.  In  some  areas,  improvement  in  the  allocation  or  resources 
across  areas  could  be  enhanced  by  having  peer  review  from  the  user  communities.  For 
instance,  policy-related  research  is  supposed  to  inform  a  variety  of  areas  of  decision 
making  by  the  government.  There  are  people  in  the  user  community  who  are  customers 
for  the  research.  If  the  group  of  people  who  bridge  the  scientific  and  user  communities, 
such  as  those  in  OSTP  and  CEA  who  are  both  researchers  and  policy  makers,  could 
become  more  involved  in  trying  to  make  these  judgments,  this  would  assist  in  the 
resource  allocation  process.  For  example,  in  certain  areas  of  the  economy  that  have 
shrunk  in  size,  spending  is  related  to  their  relative  importance  50  or  75  years  ago.  There 
are  other  sectors  of  the  economy,  like  the  service  sector,  which  have  grown  enormously, 
where  we  are  not  spending  any  money  on  research  at  all.  If  you  involved  some  policy 
analysts  they  would  point  out  that  we  are  allocating  resources  in  a  way  that  is  not 
commensurate  with  the  current  structure  of  our  economy. 

One  of  the  problems  encountered  when  discussing  critical  technologies  is  that  it  is 
easy  to  become  divorced  from  the  capability  that  is  the  desired  end-product.  For 
instance,  an  assessment  of  Navy  expenditures  would  raise  questions  about  why 
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investments  are  being  made  in  basic  research  in  biological  sciences,  mathematics,  and 
other  fundamental  domains  of  investigation.  The  answer  is  that  the  foundation  for  a 
particular  capability  may  come  from  fundamental  research  into  areas  such  as  data 
compression  algorithms  or  biological  sensors. 

Industry  Role  in  Science  and  Technology 

R&D  and  technology  development  are  fixed  costs,^  and  under  competitive 
conditions  that  raises  the  issue:  Who  is  going  to  pay  for  the  fixed  costs,  particularly  if 
they  are  costs  not  appropriable  by  the  company  carrying  out  the  R&D  investment?  In  a 
previous  era,  the  U.S.  economy  was  far  enough  ahead  of  everyone  else  that  companies 
could  afford  to  do  a  lot  of  basic  R«&D  and  follow-up  work  on  their  own.  Many  of  these 
investments  did  not  have  immediate  commercial  payoff,  but  they  were  sustained  by  the 
companies  because  they  operated  in  markets  where  they  were  somewhat  protected  by 
their  lead  in  technology.  This  is  no  longer  the  case.  Today,  most  technology  companies 
are  in  such  a  competitive  environment  that  they  are  being  pressed  to  specialize  only  in 
technologies  that  have  a  very  immediate  commercial  payoffs.  Because  of  the 
globalization  of  the  world  economy,  there  is,  in  some  sense,  an  even  greater  need  than 
previously  for  government  to  support  technology  that  has  a  general  payoff  or  is  pre¬ 
commercial  and  does  not  fit  into  the  pattern  required  to  justify  commercial  investment. 

There  is  no  substitute  for  smart  people  making  good  judgments  in  the  selection 
process.  It  is  important  to  recognize  that  decisions  are  made  not  only  to  start  programs, 
but  also  to  stop  them.  Unfortunately,  the  government  has  a  poor  record  in  stopping 
programs,  partly  because  of  the  political  process,  and  partly  because  of  a  need  for  honest 
assessments  of  markets  and  opportunities.  If  we  create  and  exercise  an  opportunity  to 
curtail  ventures  that  are  not  going  well,  then  the  pressure  on  the  first  decision  is  not  as 
great  because  corrections  can  be  made.  There  needs  to  be  more  focus  on  how  to  assess 
continually  the  status  of  projects,  and  that  means  a  need  to  involve  industry. 

Technology  Infrastructure — The  supplier  base  is  very  important  as  a  technology 
innovator.  Looking,  for  example,  at  flat  panel  displays,  the  issue  is  not  about  inventing  a 
new  display,  but  manufacturing  equipment.  To  move  firms  into  the  manufacture  of  flat 
panel  displays  requires  much  more  than  just  the  display  research  itself.  Much  of  the 
money  that  went  into  SEMATECH  helped  build  manufacturing  equipment.  A  big-ticket 
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Review  Note:  It  is  not  clear  why  R&D  and  technology  development  are  regarded  as  fixed  costs. 
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item  that  DoD  is  faced  with  right  now  is  lithography.  The  Semiconductor  Industry 
Association  (SIA)  is  looking  for  a  lot  of  help  from  the  government  to  keep  the 
lithography  business  healthy.  Without  a  good  lithography  business,  this  nation  will 
depend  on  foreign  sources  for  its  lithography  capabilities.  In  the  computer  area, 
infrastructure  includes  the  ability  to  have  design  tools.  Whether  electro-optics,  optical 
electronics,  or  micro-electromechanical  systems  (MEMS),  there  is  an  infrastructure  cost. 
While  technology  infrastructure  has  not  necessarily  been  considered  an  R&D  investment, 
it  certainly  affects  the  products  that  we  want  to  be  able  to  make. 

One  must  distinguish  between  in-house  and  outsourcing  for  technology  needs 
and  R&D.  Because  of  the  change  in  the  structure  of  R&D  and  investment  in  the  world 
community,  many  firms  are  using  their  supplier  base  to  provide  technology  that  they 
might,  otherwise,  have  pursued  in-house.  That  means  that  the  supplier  base  tends  to  be 
small-  and  medium-sized  firms  that  are  very  agile  and  have  new  and  innovative  ideas 
they  can  develop  very  quickly.  This  suggests  that  there  needs  to  be  some  thought  about 
how  to  leverage  federal  funds  that  are  spent  on  R&D  in  partnerships  with  small 
businesses,  and  how  best  to  reach  out  to  them.  In  turn,  how  do  we  assist  these  enterprises 
in  understanding  what  the  government  needs  and  how  its  needs  can  be  satisfied, 
particularly  for  dual-use  applications? 

There  appears  to  be  a  real  gap  between  the  rich  opportunities  offered  by  small 
firms  and  the  financial  resources  available  to  them  to  pursue  promising  ideas.  We  do  not 
yet  have  an  answer  as  to  how  to  fuel  innovation  by  small  firms  that  are  idea  rich  but 
capital  poor.  This  is  true  for  small  supplier  firms  as  well  in  technologically  advanced 
industries. 

Technology  must  be  integrated  into  the  workplace.  Eventually,  all  technology 
becomes  implemented  and  applied  by  users.  They  might  be  downstream  users  or 
equipment  suppliers,  but  eventually  there  is  going  to  be  some  employee  of  some 
company  who  is  supposed  to  push  a  button  or  turn  a  knob.  Current  technology  policy, 
occasionally  or  often  but  not  consistently,  is  building  on  the  best  practices  of  involving 
users  in  the  design  of  technology,  making  sure  that  we  consider  embedding  training  and 
so  forth.  This  is  an  explicit  recognition  that,  for  a  long  time,  there  has  been  a  tension 
between  the  worker  community  and  a  technology  community.  Integrating  work  and 
technology  is  not  a  new  goal  for  technology  policy.  The  notion  of  concurrent 
engineering  is  that  there  are  not  separate  technology  and  labor  concerns.  But  one  part  of 
accomplishing  technology  policy  is  making  sure  that  the  goals  of  users  are  reflected  in 
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technologies  developed  and  deployed.  We  can  improve  our  ex  ante  review  to  ensure  that 
technology  is  used  by  asking  those  seeking  investment  how  they  are  relating  to  users  and 
how  are  they  investigating  their  users’  needs. 

There  is  a  need  for  customer-driven  science  and  technology  investments.  The 
public  does  not  trust  investments  that  are  not  customer  driven.  In  industry,  and  in 
government  mission-oriented  organizations,  this  problem  is  solved  by  trying  to  see  what 
the  development  arm  wants  from  research  or,  in  the  case  of  industry,  what  the  operating 
department  wants  from  the  central  research  lab.  This  is  done  to  focus  on  customer  needs. 
In  industry,  the  ultimate  customer  is  the  public — ^the  marketplace.  This  also  relates  to 
what  might  be  termed  “design  features.”  In  the  area  of  technology,  having  firm 
participation  and  joint  ventures  is  a  way  of  assuring  that  the  process  of  selecting 
investments  is  more  customer  driven. 

The  very  fact  that  some  technology  investments  are  high  leverage  and  nonlinear 
makes  them  difficult  to  grasp  upfront,  or  even  after  the  fact.  But  these  are  exactly  the 
reasons  that  they  may  be  essential  areas  for  investment.  Often  the  role  of  research  is  to 
create  entirely  new  opportunities,  making  it  less  a  question  of  ROI  and  more  an  issue  of 
creating  a  new  reality  or  a  new  future.  This  means  there  is  a  need  to  think  about  how 
things  might  be  in  the  future. 

The  time  scales  for  developing  new  products  are  vastly  different  between  the 
defense  and  commercial  sectors.  The  government  budget  process  and  the  time  scales 
associated  with  it  drive  product  development  along  time  lines  longer  than  those  of  the 
commercial  marketplace.  The  government’s  ability  to  affect  the  commercial  sector  in  the 
near  term  is,  therefore,  best  accomplished,  through  collaborative  mechanisms  such  as 
CRADAs  and  patent  licensing.  Part  of  the  strategy  must  be  to  bring  industry  into  the 
development  process  much  earlier  because  government  investments  in  defense  have  an 
impact  on  the  next  generation  of  markets,  not  today’s.  Engaging  industry  as  a  partner  to 
government  affords  them  a  view  of  the  next  generation  of  markets. 

Defense  firms,  in  the  face  of  shrinking  markets,  should  endeavor  to  increase  the 
proportion  of  R&D  they  invest  in  commercial  activities  to  assist  in  moving  toward  a 
greater  share  of  commercial  business  in  their  portfolios.  It  would  seem  a  questionable 
investment  strategy  if  defense  firms  directed  only  a  small  percentage  of  their  resources 
toward  commercial  markets.  To  that  end,  one  could  argue  that  a  50/50  investment 
strategy  would  be  appropriate  for  the  long  term  to  realize  the  goal  of  a  more  competitive, 
affordable  defense  product.  The  likelihood  of  success  in  being  able  to  bring  technologies 
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profitably  to  the  market  is  the  limiting  factor  that  prevents  large  defense  contractors  from 
more  aggressively  moving  into  commercial  markets. 
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Session  Two 

TAILORING  METRICS  TO  ECONOMIC  POLICY  OBJECTIVES: 
THE  CASE  OF  ATP,  MEP,  AND  THE  NIST  LABORATORIES 


The  introductory  part  of  this  session  involved  an  overview  of  the  three  main 
programs  of  the  National  Institute  for  Standards  and  Technology  (NIST):  the  Laboratory 
Program  (Labs);  the  Manufacturing  Extension  Program  (MEP);  and  the  Advanced 
Technology  Program  (ATP).  The  session  then  moved  into  a  general  discussion  of  how 
public  sector  investments  in  commercially  relevant  technologies  should  be  evaluated. 

THE  NATIONAL  INSTITUTE  FOR  STANDARDS  AND  TECHNOLOGY 

Laboratory  Program 

The  traditional,  core  program  at  NIST  is  the  laboratory  effort.  This  work  was 
organized  and  begun  as  the  National  Bureau  of  Standards  (NBS)  in  1901.  In  1988, 
Congress  changed  the  name  of  the  organization  to  the  National  Institute  for  Standards 
and  Technology  to  reflect  their  growing  mission. 

The  NIST  laboratory  program  serves  as  the  nation’s  measurement  laboratory.  It 
provides  the  technology  infrastructure  that  supports  the  “business  of  business.”  The 
mission  reflects  the  needs  for  widely  accepted  standards  which  began  with  interchange¬ 
able  parts  during  the  Industrial  Revolution  a  century  ago.  Worldwide,  there  are  many 
similar  institutions,  established  by  other  governments  and  through  international 
agreements  to  pursue  common  standards  to  facilitate  trade,  production,  evaluation,  and 
quality. 

The  work  of  the  NIST  laboratories,  today,  is  very  broad.  Its  constituency  includes 
mature  and  traditional  industries  involved  in  manufacturing,  materials,  chemicals,  and 
petroleum.  More  recently,  information  technologies  and  electronics  have  emerged  as 
areas  of  particular  importance.  Laboratories  also  keep-up  with  emerging  new  industries 
such  as  bio-technology  and  advanced  materials.  The  mission  of  the  laboratories  is  to 
pursue  standards  and  measurement  techniques  that  no  one  company  can  do  for 
themselves,  but  from  which  every  company  in  an  area  would  benefit. 
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One  area  where  there  is  a  very  strong  commitment  today,  as  in  the  past,  is  that  of 
metrology — ^the  science  and  technology  of  measurement.  Metrology  has  applications  in 
the  areas  of  semiconductors  and  microelectronics.  Recognition  of  the  importance  of  this 
work  to  the  microelectronic  industry  is  found  in  the  recent  technology  road  map  put 
together  by  the  Semiconductor  Industry  Association  (SIA).  One  piece  of  that  road  map  is 
a  recognition  of  the  role  of  measurement  science  and  technology  in  the  business  of 
semiconductor  manufacturing. 

It  is  apparent  that  semiconductor  technology  depends  on  extremely  precise 
measurements,  when  one  considers  the  feature  sizes  of  modem  semiconductor  circuits. 
Annually,  NIST  spends  approximately  $40  million  in  its  electronics  laboratory  in  the  area 
of  measurement  and  technology  infrastructure.  Given  the  pivotal  role  of  semiconductors 
in  the  overall  structure  of  the  national  economy,  these  laboratory  investments  are  highly 
leveraged. 

Manufacturing  Extension  Partnerships 

The  goal  of  this  program  is  to  establish  a  national  network  of  centers  that  can 
reach  out  and  provide  access  to  information  and  expertise  to  small  manufacturers  around 
the  U.S.  It  is  based  on  a  recognition  that  half  of  the  nation’s  manufacturing  capacity  is  in 
firms  of  500  or  fewer  employees.  These  small  firms  tend  not  to  be  particularly  well 
linked  to  advances  in  manufacturing  technologies  and  advances  in  new  kinds  of 
manufacturing  business  practices.  The  point  of  the  program  is  to  improve  this  access. 

There  are  two  basic  factors  to  be  considered  in  allocating  resources  in  the  MEP 
program.  One  has  to  do  with  the  nature  of  user  needs.  When  this  program  began,  there 
was  considerable  enthusiasm  for  taking  technologies  out  of  the  national  labs  and  making 
them  available  to  the  machine  shop  down  the  road.  It  turns  out  that  if  you  actually  go  to 
the  machine  shop  down  the  road  and  ask  them  what  they  need,  the  last  thing  they  mention 
is  advanced  technologies  from  the  national  laboratories.  After  considerable  effort  to 
understand  what  the  customers  really  needed,  the  program  today  focuses  on  more 
practical  technology  applications.  The  program  successfully  helps  companies  to 
implement  just-in-time  inventory  systems,  assists  with  improving  the  flow  of  work  on  the 
shop  floor,  identifies  the  kinds  of  training  techniques  that  will  help  small  firms  integrate 
new  technologies  into  their  production  systems,  and  so  forth.  Once  in  a  great  while,  a 
technology  problem  does  arise  that  can  be  solved  by  a  national  laboratory  or  a  university, 
but  this  is  not  the  major  focus  of  the  program. 
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The  second  factor  to  be  considered  is  this:  How  do  you  actually  build  a  national 
network  consisting  of  100  centers,  each  of  which  is  supported  on  a  cost  shared  basis  with 
a  state  or  local  government?  That  is  the  top-down  picture.  The  way  the  network  is 
actually  being  constructed  is  by  experienced  private  sector  engineers  who  operate  out  of 
the  centers  and  visit  the  firms  in  their  area.  In  a  very  practical,  hands-on  way  these 
engineers  look  at  the  various  firms  to  see  how  they  go  about  their  manufacturing  process. 
They  also  look  at  how  their  business  is  operating,  and  help  those  companies  identify 
technologies  to  make  a  difference  in  their  competitive  capabilities. 

Much  progress  has  been  made  towards  establishing  the  national  system.  Today 
there  are  forty-four  centers  in  thirty-two  states.  These  are  not  discrete  centers  operating 
on  their  own.  They  form  part  of  a  network  with  information  shared  across  state 
boundaries,  with  linkages  to  universities  and  laboratories  around  the  country,  and  with 
organizations  such  as  the  EPA,  the  Department  of  Labor,  and  the  Small  Business 
Administration. 

MEP  centers  are  established  on  the  basis  of  a  competition.  Winning  proposers  are 
offered  an  opportunity  to  have  their  concept  for  a  center  cost  shared  by  the  federal 
government.  When  proposals  come  in  for  extension  centers,  the  factors  considered  are: 
“Does  this  center  really  understand  the  manufacturers  in  its  area?  Do  they  understand  the 
manufacturers’  needs?  Have  they  coordinated  with  other  service  providers  in  the  area  so 
that  they  are  not  just  proposing  a  limited,  point  solution?” 

Although  there  is  a  goal  of  national  coverage,  geography  is  not  considered  to  be  a 
primary  selection  criterion.  Rather,  there  is  a  deliberate  strategy  to  make  sure  that  some 
access  for  all  manufacturers  around  the  country  is  established,  and  this  is  being 
accomplished  by  working  with  States  that  have  not  yet  been  awarded  centers. 

The  Advanced  Technology  Program  (ATP) 

The  goal  of  the  ATP  is  to  improve  U.S.  economic  growth  and  competitiveness 
through  selective,  public-sector  technology  investments  that  are  commercially-relevant. 
ATP  resources  are  allocated  according  to  strategic  technology  areas.  The  canonical 
model  is  to  invest  approximately  $150  million  in  a  technology  area  over  a  5-year 
period — a  limited  investment.  ATP  allocates  funding  so  there  is  a  strong  likelihood  of  a 
dramatic  advance  in  each  of  the  areas. 

Because  national  economic  growth  and  performance  depends  on  industry,  the 
ATP  selects  its  investment  areas  from  a  set  provided  by  industry.  A  top-down  selection 
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process  is  neither  feasible  nor  efficient.  The  best  places  to  allocate  funding  are 
identifiable  through  a  bubble  up  process  where  ideas  come  in  and  are  considered  against 
a  set  of  clear  criteria.  To  quote  David  Barrymore,  Deputy  Secretary  at  Commerce:  “It  is 
not  an  epiphany,  it  is  an  engineering  process.”  It  is  not  about  finding  the  single  best  thing 
in  the  entire  economy  to  do,  it  is  about  finding  some  excellent  things  to  do. 

The  process  begins  with  ideas  coming  in  from  the  technical  community,  primarily 
industry,  but  it  is  open  to  the  broader  technical  community.  Four  criteria  are  used  to 
evaluate  the  ideas  received: 

•  Potential  for  U.S.  economic  benefit.  From  a  systems  perspective,  if  ATP 
undertakes  a  technology  program,  and  if  it  is  successful,  how  will  that 
translate  into  economic  benefit?  How  will  the  companies  that  are  involved 
benefit?  How  will  the  technologies  that  they  create  help  their  customers? 
How  will  it  change  the  relationship  with  their  suppliers?  The  issue  of 
whether  or  not  the  technologies  will  ever  get  implemented  is  extremely 
critical. 

•  Strength  of  the  technical  ideas.  There  are  many  instances  where  there  are 
marvelous  economic  opportunities,  but  the  barriers  to  achieving  them  do  not 
have  to  do  with  technology.  In  such  cases  ATP  is  not  the  appropriate  tool. 
ATP  is  looking  for  places  where  technical  work  can  start  changing 
opportunity.  The  program  therefore  searches  for  brilliant  technical  ideas, 
nonlinear  ideas  that  create  a  new  kind  of  a  reality  as  a  result  of  their 
innovation. 

•  Strong  industry  commitment.  Is  there  enough  commitment  to  complete  the 
program?  For  example,  there  are  those  who  will  write  proposals  merely  to 
secure  cost  sharing.  Is  a  proposer  committed  to  taking  the  next  step  in 
implementing  the  technologies,  and  then  to  taking  them  out  to  the 
marketplace? 

•  High  leverage  opportunities.  How  can  ATP  make  a  big  difference?  It  is 
important  not  to  duplicate  other  efforts.  But  most  important,  the  program 
looks  for  places  where  small  investments  result  in  big  returns. 

To  date  ATP  has  completed  two  cycles  of  its  industry-interactive  process  in  about 
a  year  and  a  half  This  has  resulted  in  approximately  1,000  white  papers  coming  in  from 
industry  for  comparison  with  the  four  criteria. 

The  white  papers  are  grouped  according  to  technology  categories,  and  are  then 
compared  with  active  programs  in  other  agencies  of  the  federal  government.  To  make 
sure  that  duplication  of  effort  is  avoided,  white  papers  are  reviewed  outside  of  NIST  as 
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well.  Where  there  appears  to  be  significant  industry  interest  and  no  possibility  of 
duplication  of  effort,  workshops  with  industry  are  held  to  more  completely  investigate 
competitive  areas. 

ATP  has  made  a  total  of  177  awards.  Half  of  those  awards  have  been  won  by 
small  companies  or  joint  ventures  led  by  small  companies.  The  process  is  available  and 
open  for  small  firms. 

ROUND  TABLE  DISCUSSION 

The  NIST  Laboratories 

Measurement  infratechnologies  are  one  class  of  infrastructure  that  NIST  provides 
to  industry.  These  types  of  measurement-intensive  tools  and  data  are  found  at  all  of  the 
major  stages  of  the  typical  technology-based  industry.  For  instance,  in  the  area  of  R&D 
materials  characterization,  data  bases  are  essential  now  that  scientists  and  engineers  are 
moving  atoms  around  in  materials  in  order  to  change  performance  attributes  and 
customize  products.  You  have  to  be  able  to  measure  the  results  of  an  experiment  in  order 
to  convince  corporate  management  that  you  have  achieved  an  experimental  objective. 
Once  you  reach  the  point  of  commercialization  and  production,  you  have  very 
measurement  intensive  process  models  that  are  applied  in  order  to  achieve  quality 
objectives  to  increase  yield.  Finally,  in  the  market  stage,  technologically  complex 
products  have  a  number  of  critical  performance  attributes.  There  are  often  disputes 
between  the  buyer  and  the  seller  as  to  how  to  measure  these  attributes  for  purposes  of 
product  acceptance. 

The  typical  laboratory  inframetrics  project  at  NIST  is  relatively  small.  Each 
project  typically  supports  a  large  number  of  applications.  The  impact  of  these  small 
investments  can  be  quite  significant  in  the  aggregate.  To  measure  the  impacts  of  such 
laboratory  investments,  investigators  typically  sit  down  with  the  laboratory  technical 
people  and  their  industry  counterparts  in  order  to  translate  the  technical  output  of  the 
work  at  NIST  into  economic  impacts.  Economics  is  a  foreign  language  to  the  typical 
scientist  or  engineer,  and  so  it  is  not  a  straightforward  process.  Questions  include  the 
area  of  impact:  Is  it  R&D?  Is  it  production?  Is  it  marketing?  Often  it  is  more  than  one 
area.  The  typical  NIST  project  spans  5  to  10  years,  so  it  is  possible  to  collect  a 
reasonable  time  series  of  data  on  the  cost  to  NIST  of  conducting  the  research,  the  cost  of 
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technology  transfer,  the  cost  of  assimilation  by  the  industrial  firms,  and  the  benefit  stream 
that  occurs. 

Because  these  projects  and  costs  are  bounded  in  time,  the  internal  rate  of  return 
method  can  be  used  to  measure  or  estimate  the  net  benefit  of  a  particular  investment  by 
NIST.  The  term  “social  rate  of  return”  is  the  economist’s  version  of  what  the  business 
literature  calls  the  “internal  rate  of  return.”  Scientists  and  engineers  tend  not  to  like  the 
term  “social  rate  of  return,”  so  NIST  calls  this  the  “spill-over  rate  of  return.”  In  the 
Mansfield  type  of  case  studies,  internal  rate  of  return  is  used  as  a  metric  for  measuring 
the  rates  of  return  to  a  private  investment  in  technology.  NIST  calls  the  internal  rate  of 
return  of  the  Mansfield  type  the  “innovator  rate  of  return.” 

From  the  perspective  of  national  technology  infrastructure,  one  is  not  concerned 
with  the  innovator  rate  of  return  because  a  government  laboratory  invests  in  the  general 
public  return  of  the  industrial  technology.  Therefore,  what  you  measure  is  the  spill-over 
rate. 

To  date,  NIST  has  looked  in  detail  at  eleven  laboratory  projects.  Four  have  been 
done  in  the  semiconductor  area,  so  a  broader  picture  is  beginning  to  emerge  of  how 
technology-related  research  at  NIST  results  in  spill-over  rates  of  return.  NIST  is  also 
branching  out  into  related  areas  such  as  information  technology,  looking  into  areas  such 
as  optical  fibers. 

The  output  of  laboratory  research  can  be  a  new  procedure  or  a  new  data  base,  but 
much  ends  up  being  embodied  in  standards.  For  example,  in  the  case  of  optical  fiber, 
NIST  contributed  the  technical  basis  for  22  industry  standards  over  the  decade  of  the 
1980s.  A  median  rate  of  return  of  167  percent  was  calculated  as  the  impact  of  these 
standards.  By  comparison,  the  median  rate  of  return  found  by  Mansfield  and  subsequent 
NSF-sponsored  studies  of  private  sector  innovations  was  76  percent.  In  a  more  recent 
Mansfield  study,  returns  to  industry  from  sponsoring  academic  research  were  found  to  be 
28  percent.  Note  that  the  general  rates  of  return  to  capital  in  the  U.S.  economy  are 
around  13  or  14  percent. 

There  are  some  caveats  because  to  this  point  NIST  has  chosen  eleven  projects 
which,  through  some  screening,  have  met  specific  criteria  for  an  evaluation.  For 
example,  there  was  accessible  data  and  sufficient  time  had  passed  that  there  was  some 
reasonable  benefit  stream  to  relate  to  the  costs.  The  feasibility  of  doing  such  an  impact 
study  varies  from  one  technical  area  to  another,  depending  on,  among  other  things,  the 
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distribution  of  benefits  across  industries.  There  is  a  certain  amount  of  selection  bias  in 
the  projects  that  NIST  has  evaluated.  It  is  going  to  take  several  more  years  to  accumulate 
enough  individual  studies  so  that  we  may  feel  more  confident  in  making  generalizations 
about  expected  rates  of  return  from  this  type  of  infrastructure  support.  Based  on  results 
to  date,  an  economist  would  say,  “Well,  you  have  a  negatively  sloped  marginal  efficiency 
investment  curve,  so  these  high  returns  mean  we  are  way  underfunded.”  That  is  an 
argument  that  our  people  love  to  hear  and  would  like  to  plaster  on  every  wall  of 
Congress. 

Preliminary  evidence  thus  indicates  that  investments  in  these  infrastructure 
programs  are  extremely  effective.  The  interactions  between  industry  and  NIST,  in 
identifying  needs,  agreeing  upon  a  program,  carrying  it  out,  and  then  transferring  the 
results  are  very  impressive.  In  many  cases,  the  output  of  research  is  transferred  and  used 
before  the  industry  ever  gets  around  to  adopting  it  as  a  standard.  So  although  standards 
are  important  to  the  economy,  certain  types  of  NIST’s  public  sector  infrastructure  support 
get  used  before  they  become  promulgated  as  a  standard. 

In  summary,  the  types  of  quantitative  impacts  that  NIST  typically  measures 
include  things  such  as  quality,  including  variability  in  a  particular  attribute  of  a  product, 
and  reliability.  NIST  is  also  concerned  with  measuring  productivity  in  terms  of  yield  or 
overall  cost  efficiency  in  the  production  process  within  an  industry. 

Time  to  market  is  another  impact  that  NIST  frequently  measures.  Studies  try  to 
capture  market  share  impacts  as  much  as  possible,  but  market  share  is  a  difficult  metric  to 
attribute  to  the  NIST  work,  even  in  part. 

Studies  must  account  for  transaction  costs.  These  include  the  costs  incurred  by 
companies  in  resolving  disputes,  for  example,  sending  engineers  back  and  forth  to  try  to 
agree  on  how  to  measure  the  signal  loss  rate  in  an  optical  fiber  or  the  electrical  resistance 
of  a  semiconductor  chip.  These  are  real  costs,  just  like  production  costs,  that  add  to  the 
price  of  the  product.  They  slow  down  the  diffusion  of  technology  into  the  marketplace. 
If  other  countries  are  better  organized  and  provide  their  infrastructure  more  efficiently,  it 
gives  them  a  competitive  advantage.  Transaction  costs  are  not  particularly  well  known, 
even  to  economists,  but  they  are  extremely  important  in  terms  of  the  contributions  that 
NIST  makes  in  its  infrastructure  work. 

Qualitative  information  can  be  as  important  as  quantitative  information  in  impact 
studies.  Qualitative  information  complements  quantitative  estimates  and  provides  a 
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broader  and  more  complete  picture  of  the  contributions  that  NIST  makes.  For  example, 
NIST  often  talks  to  customers  who  are  one  layer  down  the  chain,  from  the  direct 
beneficiaries.  Typically  there  is  no  attempt  to  make  quantitative  estimates  beyond  the 
primary  industry,  because  the  farther  down  that  value  added  chain  one  goes,  the  more 
other  factors  come  into  play,  and  the  more  difficult  it  is  to  isolate  the  contribution  that 
NIST  makes. 

The  interaction  between  NIST  and  industry  affects  R&D  investment  decisions. 
Simply  the  ability  to  measure  something  in  a  laboratory  can  lead  to  new  directions  in 
research  that  were  not  possible  before. 

Manufacturing  Extension  Partnerships  (MEP)  Program 

What  is  the  importance  of  technology  infrastructure  if  manufacturing  is  a 
declining  sector  in  the  national  economy?  A  coast-to-coast  flight  in  the  United  States 
takes  about  four  hours,  during  which  there  are  about  2.5  hours  of  agriculture  along  one’s 
route.  Yet  the  agricultural  sector  of  the  national  economy  has  been  a  declining  one  for 
years.  When  we  try  to  assign  a  value  to  the  importance  of  agriculture  to  the  economy,  the 
problem  becomes:  How  do  we  count?  Do  we  count  farmers,  do  we  count  the  big  animal 
veterinarians,  or  do  we  count  the  companies  that  are  providing  fertilizer  so  that  there 
aren’t  so  many  farmers  needed.  In  fact,  we  have  to  look  at  the  agricultural  complex. 

In  the  same  way,  to  understand  the  economics  of  manufacturing,  we  have  to  look 
at  the  manufacturing  complex.  In  both  cases,  it  is  important  to  understand  the  supporting 
linkages  in  the  economy  which  allow  agriculture  and  manufacturing  to  represent  smaller 
and  smaller  portions  of  the  overall  economic  picture.  When  all  of  manufacturing  is 
accounted  for  in  this  way,  to  include  activities  upstream  from  the  shop  floor,  the 
percentage  of  the  economy  that  was  involved  in  manufacturing  is  vastly  larger  than 
appears  on  the  surface.  Infrastructure  is  the  key  to  success  in  maintaining  these  upstream 
linkages  in  the  U.S.  manufacturing  economy.  The  issue  is  that  the  U.S.  is  less  good  than 
other  nations  at  diffusing  infrastructure  technology.  As  a  consequence,  firms  tend  to 
move  production  activities  offshore.  The  question  here  is:  What  kind  of  infrastructure 
do  we  need  so  that  we  can  take  full  advantage  of  linkages?  The  analogy  to  agriculture  is, 
probably,  a  pretty  good  one — we  have  remained  pretty  competitive  in  that  field. 

MEP  wants  to  emulate  the  success  of  agricultural  extension.  But  the  program 
does  not  want  to  look  like  the  agriculture  one.  If  manufacturing  will  represent  only  2 
percent  of  the  U.S.  economy,  it  will  not  be  desirable  to  have  exactly  the  same  structure  in 
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place  when  it  was  19  percent.  If  the  small  manufacturing  segment  continues  to  decline, 
then  one  wouldn’t  want  manufacturing  extension  services  in  every  county  in  the  United 
States.  There  is  a  need  to  size  the  program  to  the  number  of  customers.  Moreover,  if  you 
are  funding  the  same  firms  and  projects  after  10  or  20  years,  something  is  not  working. 
A  key  issue  is  to  make  sure  that  the  best  activities  receive  funding.  At  the  moment,  MEP 
legislation  has  a  sunset  clause  that  terminates  federal  funding  at  the  end  of  6  years.  This 
has  provoked  considerable  discussion  about  what  a  sunset  really  implies,  and  whether  the 
legislation  should  be  changed  to  continue  the  program.  The  real  question  is  how  to  make 
sure  the  program  structure  remains  sufficiently  flexible  to  keep  up  with  changing 
business  environment. 

What  lessons  has  the  MEP  learned  from  other  countries?  Other  countries  have 
benefited  from  having  a  sustained  infrastructure  program  of  the  MEP  kind.  Certainly,  the 
European  countries,  and  Japan,  Korea,  and  Taiwan  have  all  benefited  from  having  this 
type  of  infrastructure  activity.  These  may  provide  a  model  for  new  kinds  of  American 
programs,  for  example,  the  Fraunhofer  Institutes,  which  are  applications-oriented  R&D 
centers  in  Germany.  There  are  other  possibilities,  if  we  are  smart  enough  and  attentive 
enough  to  keep  these  institutions  focused  on  the  need. 

How  does  industry  view  the  MEP  program?  The  MEP  is  considered  an  extremely 
valuable  program.  It  is  looked  upon  as  being  a  communications  program — how  to 
communicate  best  current  practices  among  industries.  It  is  interesting  to  note  that  MEP  is 
not  centered  around  universities,  but  instead  is  centered  around  the  factory  floor  and 
guided,  to  a  great  degree,  by  the  needs  of  factory  employees.  This  makes  for  appropriate 
assistance  for  small  companies.  Big  companies  can  afford  to  find  out  what  each  other  is 
doing.  They  can  afford  to  go  to  Japan  and  find  out  what  is  happening.  Small  companies 
cannot.  We  need  both  big  eompanies  and  small  companies  working  together  in 
supplier/customer  relationships.  Therefore,  it  is  in  the  long  term  interest  of  both  small 
companies  and  big  companies  that  all  of  U.S.  industry  gets  access  to  the  best  current 
practices.  MEP  needs  to  be  in  place  for  a  long  term,  even  though  it  will  have  to  change 
with  the  nature  of  industry.  Even  if  we  drive  industry  down  to  three  percent  of 
employment,  which  should  be  viewed  as  a  success,  then  it  will  still  be  as  important  to  the 
nation  as  agriculture. 

Who  are  the  customers  of  small  firms?  Large  companies  who  want  to  become 
world  class  or  remain  world  class  are  investing  a  lot  in  their  supplier  chains.  They  want 
their  suppliers  to  have  six  sigma  quality,  or  Just  in  time  delivery,  or  engage  in  electronic 
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commerce.  Each  of  those  large  companies  has  some  rather  specific  requirements  that 
they  are  imposing  on  smaller  firms. 

What  steps  are  being  taken  in  the  MEP  to  use  information  from  larger  companies 
to  assess  the  progress  of  effectiveness  of  MEP  with  small  companies?  MEP  is  starting  to 
look  at  supplier  chains.  It  is  looking  at  return  on  investment  to  larger  companies  in  the 
supplier  chain.  Sometimes,  in  a  price-competitive  situation,  the  benefit,  doesn’t  accrue  to 
the  smaller  firms  but  to  the  larger  firms  they  supply,  or  to  the  ultimate  consumer. 

How  does  the  government  know  when  to  get  out  of  the  manufacturing  extension 
business?  This  is  an  important  question  since  the  agricultural  extension  program  has 
been  operating  since  1862.  The  concern  is  that  government  initiates  such  programs  and 
creates  a  large  infrastructure — a  legacy  of  100  centers — which  are  supported  by 
constituencies  that  are  very  difficult  to  turn  off  As  long  as  the  government  is  prepared  to 
fund  such  programs,  there  is  no  impetus  for  them  to  become  self-sustaining. 

Evaluation  of  the  MEP  requires  both  short  and  long  run  metrics.  In 
manufacturing  extension,  NIST  is  building  an  evaluation  system  to  go  along  with  the 
program.  There  is  an  up-front  activity  of  interviewing  companies  to  whom  the  program 
has  provided  services  to  understand  their  assessment  of  what  benefits  it  offers.  This  also 
includes  a  reflection  of  customer  satisfaction. 

The  second  part  of  the  evaluation  is  to  go  back  and  look  at  companies  served  to 
see  if  their  projections  were  realized.  The  kinds  of  successes  for  companies  include 
things  like  reduced  scrap  rate,  shorter  product  cycle  times,  the  ability  to  see  new  market 
opportunities,  and  productivity  enhancements.  These  are  observable  within  a  few  years 
of  rendering  assistance.  They  are  also  fairly  concrete  outcomes  which  are  projectible, 
and  are  measurable  after  the  fact. 

Many  different  types  of  data  come  in  from  MEP  centers  around  the  country. 
Early  results  reflect  the  work  of  44  centers  which  served  approximately  12,000  firms 
around  the  country.  Of  course  this  must  be  compared  to  the  380,000  small  manufacturers 
operating  in  the  US.  The  projected  benefits  to  companies  from  MEP  assistance  appears 
to  be  on  the  order  of  an  8-fold  benefit  for  each  federal  dollar  expended.  That  means  for 
each  federal  dollar  that  has  been  spent  on  the  program,  projections  from  these  companies 
indicate  that  they  will  see  about  eight  dollars  in  economic  benefits  from  business 
operations.  Of  course  there  needs  to  be  a  much  richer  and  a  much  more  robust  evaluation 
to  verify  these  claims. 
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Advanced  Technology  Program  (ATP) 

International  competition  in  technology  is  central  to  trade  promotion  and  a 
healthy  balance  of  payments.  We  are  competing  in  a  world  market.  Industrial 
competition  involves  more  than  firm  A  or  firm  B  in  the  United  States.  We  must  look  at 
the  issue  in  terms  of  firms  A  and  B  in  the  U.S.,  firm  C  in  Japan,  firm  D  in  Germany,  and 
so  forth.  The  U.S.  is  a  large  country  whose  internal  market  has,  in  the  past,  allowed  us  to 
disregard  some  of  the  competitive  processes.  The  countries  where  competitive  processes 
move  most  swiftly  are  small  countries  like  Denmark,  where  there  is  no  choice 
whatsoever.  Since  the  mid- 19th  century  the  Danes  have  put  in  place  programs  to 
advance  their  industries  into  more  sophisticated,  high  value  added  niches  in  world 
markets. 

This  has  been  a  wise  strategy.  For  example,  at  one  time  Polish  hams  were  starting 
to  sweep  into  European  markets.  The  Danish  response  was  to  ask:  “Why  would  anyone 
want  to  buy  a  Polish  ham  instead  of  a  high  value  added  Danish  ham?”  The  Danes  proved 
to  be  right,  because  the  French  bought  the  Polish  hams  and  shipped  them  to  Russia, 
which  is  where  the  Poles  had  been  shipping  them  in  the  first  place.  So  in  the  end  the  real 
threat  to  the  Danes  was  in  their  heads,  it  was  not  Polish  hams.  The  rest  of  Europe 
adopted  the  standards  that  they  were  adopting,  and  the  Danes  adopted  the  technology 
support  programs  that  gave  their  nation  an  advantage.  The  real  threat  was  not  hams  from 
Poland  but  the  successful  implementation  of  similar  infrastructure  programs  and  similar 
standards  elsewhere  in  Europe. 

Who  is  leveraging  whom?  There  is  considerable  talk  today  about  leveraging 
federal  government  funds.  On  the  industry  side  they  talk  about  leveraging  their  own 
funds  with  federal  dollars.  Collaborative  programs  are  apparently  of  general  benefit. 
Partnerships  imply  overlapping  objectives  so  that  there  is  a  real  mutual  benefit.  This  is 
described  in  the  business  world  as  a  “win-win  situation.”  Government  is  leveraged 
because  it  is  working  with  industry,  and  industry  is  leveraged  from  federal  dollars. 

Is  AT&T,  for  instance,  big  enough  and  rich  enough  to  make  such  investments  on 
its  ownl  The  issue  of  the  size  of  a  company  is  one  that  comes  up  frequently.  There  has 
been  much  discussion  about  how  and  where  industry  is  spending  its  R&D  dollars.  There 
is  an  inside-the-beltway  fiction  that  companies  with  over  500  employees  don’t  innovate, 
or  don’t  need  federal  assistance.  Clearly  in  the  case  of  firms  such  as  Motorola,  AT&T, 
GE,  and  others,  this  is  patently  not  the  case.  However,  large  companies  are  decreasingly 
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spending  their  R&D  funds  in  areas  which  are  likely  to  produce  socially  and  economically 
beneficial  spill-overs. 

What  is  the  impact  on  firms  not  assisted  by  the  ATP  from  the  introduction  of  new 
technologies  which  receive  support  from  the  government?  When  you  are  developing  new 
technologies  you  are  also  displacing  older  technology.  When  company  A  owns  the  older 
technology,  and  company  B  has  the  newer  technology,  and  the  federal  government  is 
supporting  B,  it  is  seen  as  putting  A  out  of  business.  In  the  ATP  program,  this  issue  was 
encountered  in  the  field  of  composites.  When  the  ATP  started  its  composite  structures 
program,  it  immediately  began  to  hear  from  the  steel  industry  about  how  sinful  this  was 
because  the  applications  are  in  things  where  steel  had  once  been  used.  An  important  part 
of  the  answer  to  the  steel  industry  was  that,  if  they  wanted  to  participate  in  the  program 
they  should  send  in  their  own  technical  ideas  and  show  up  at  the  workshop  to  hear  the 
selection  criteria.  In  fact,  no  one  from  the  steel  industry,  or  any  other  industry,  indicated 
that  the  selection  criteria  were  flawed  or  incorrect.  Either  firms  see  advantage  to 
participating  in  the  program  or  they  steer  clear  of  it.  One  of  ATP’s  new  programs 
launched  in  the  fall  is  materials  processing  for  heavy  manufacturing.  Its  participants 
include  steel  manufacturers.  They  did  come  to  the  table,  and  they  did  play. 

How  do  you  capture  the  potential  loss  from  competitors  of  firms  that  are  helped? 
Does  this  mean  that  the  actual  benefit  to  the  economy  is  considerably  less  than  what  is 
attributable  to  the  immediate  beneficiaries  when  displacement  costs  are  taken  into 
consideration?  NIST  programs  result  in  companies  that  are  leaner,  meaner,  more 
aggressive,  have  bigger  market  share,  and  so  on.  They  do  better  compared  to  their 
foreign  competitors.  Increasing  quality  is  essential  to  being  a  global  competitor.  If  the 
near  term  effect  is  that  a  company  displaces  and  takes  away  orders  that  a  less  competitive 
firm  elsewhere  was  getting,  this  is  still  good  in  the  long  term  because  it  leads  to  a  more 
competitive  manufacturing  base — and  that  is,  really,  what  we  are  trying  to  get  to.  If  one 
is  not  improving  competitive  capabilities,  one  may  simply  be  shifting  locations  but  not 
improving  competitiveness. 

ATP  uses  different  time  frames  for  evaluation.  ATP  very  formally  segregates  the 
short  to  intermediate  term.  None  of  the  metrics  during  the  first  3  to  5  years  of  a  project 
involve  private  rate  of  return  calculations,  and  certainly  not  social  rate  of  return 
calculations.  Issues  addressed  during  this  period  include:  “Are  the  performers  doing 
what  they  said  they  were  going  to  do  in  the  proposal?”  “Are  they  really  doing  an 
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integrated  business  commercialization  planning  process  while  they  are  doing  their 
technology  development?” 

Short-term  metrics  are  indicators  of  success:  “Are  they  really  investing  new 
money  in  the  technology  themselves?”  “Are  they  attracting  industry  partners?”  “Are 
there  user-customer  alliances  being  formed?”  “Is  private  investor  capital  being  attracted 
into  this  technology  pool  now  that  would  not  have  been  there  without  the  ATP  laying  the 
ground  work  and  reducing  the  risk.”  “Are  jobs  being  created?”  “Are  they  really  thinking 
through  the  marketing  strategy  and  what  it  takes  to  scale  up?” 

Longer  term  metrics  incorporate  the  data  gathering  built  into  cooperative 
agreements,  a  minimum  of  every  2  years  for  6  years  after  the  project  ends.  That  is  still 
not  quite  enough,  but  it  is  about  all  that  NIST  could  get.  Hopefully,  researchers  will  be 
able  to  go  back  later  and  capture  some  more.  In  the  long  term  NIST  will  be  going  back 
regularly  and  routinely  to  do  case  studies.  That  data  gathering,  supplemented  by 
additional  broad  industry  studies  by  academics  can  help  to  ascertain  social  rates  of  return, 
gross  or  macroeconomics  effects  on  jobs  and  output,  and  GDP  output. 

The  complete  results  from  the  ATP  will  not  be  known  in  the  short  run.  The  short¬ 
term  operational  question  is  whether  or  not  the  program  is  functioning  efficiently.  The 
following  information  is  currently  available  for  preliminary  evaluations: 

•  Assessments  of  the  ATP’s  own  critical  operational  activities 

•  Portfolio  profiles  of  applicants,  recipients,  technologies,  and  projects 

•  Evaluation  of  industry’s  implementation  of  both  the  R&D  and  business 
components  of  ATP  projects 

•  Tracking  of  short-term  and  intermediate  project  results 

•  Measurement  of  long-term  economic  impacts 

There  is  considerable  data  for  the  first  three  items  and  some  data  for  the  fourth. 
There  are  only  a  very  small  number  of  projects  in  ATP  that  have  actually  been 
completed,  since  this  program  began  with  Fiscal  1990  appropriations. 

An  example  of  an  apparently  highly  successful  ATP  investment  was  that  of  a 
small  manufacturing  company  owned  and  operated  by  only  two  individuals.  They  quite 
literally  had  a  concept.  Nothing  had  been  built  or  demonstrated.  Because  of  that,  they 
were  unable  to  attract  private  capital.  ATP  did  a  very  brief  cost-share  project  with 
them — 18  months.  They  were  then  able  to  demonstrate  a  laboratory  prototype.  Based  on 
this  prototype  they  have  now  licensed  their  technology  and  lined  up  private  capital  to 


51 


continue  the  development.  The  firm  now  employs  35  and  has  a  commercialization 
pathway  to  go  forward. 

The  benefits  from  an  ATP  project  continue  long  after  the  project  is  completed. 
They  continue  to  flow  after  the  private  sector  has  taken  the  technology  to  the  marketplace 
using  private  dollars,  as  customers  start  getting  the  benefit  from  the  technology.  The 
short  term  and  the  intermediate  project  results  include  the  immediate  job  growth  that  has 
taken  place.  It  is  not  the  full-blown  effect,  but  it  is  one  piece  of  what  is  important  so  far. 
The  fact  that  firms  have  strategic  alliances  and  licensing  agreements,  and  the  fact  that 
they  have  been  able  to  attract  private  capital,  are  also  measures  of  progress. 

Are  metrics  attainable?  What  needs  to  be  put  in  place  in  order  to  get  at  useful 
measures  of  effectiveness? 

•  ATP  receives  quarterly  reports  by  which  it  tracks  technical  progress,  and 
changes  in  business  behavior,  how  targets  change,  and  how  different 
strategic  alliances  grow  or  move.  Routinely  and  quarterly,  ATP  tracks  the 
global  strategies  for  commercializing  the  technology  in  the  longer  run,  as 
earlier  planning  is  amended. 

•  Other  ATP  issues  which  are  tracked  include  plans  for  licensing,  for 
manufacture,  and  for  strategic  alliance  with  other  companies.  What  are  the 
opportunities  to  be  pursued  with  this  technology  in  the  long  run?  Has  the 
firm  started  investing  in  production  downstream?  Has  it  started  doing  market 
analysis?  Is  it  looking  at  what  competitors  are  doing?  Has  there  been  a 
change  in  competitive  advantage  in  the  marketplace  since  starting  this 
project? 

•  Then  every  year  ATP  looks  at  questions  like  these:  Have  you  attracted 
financing?  Have  you,  actually,  formed  strategic  alliances?  How  have  they 
been  helpful?  Have  you  started  to  have  spin-off  products?  Have  you  started 
earning  license  fees?  How  much  have  you  earned?  With  the  help  of 
academia,  the  ATP  is  also  developing  a  longer  term  model  to  capture  the 
spill-over  effects. 

What  are  the  time  horizons  for  assessing  program  results?  In  the  case  of  the 
ATP,  it  will  be  many  years  before  the  full  economic  effects  can  be  discussed.  In 
manufacturing  extension  we  are  closer  to  seeing  results,  but  the  program  is  still  too  young 
to  make  any  definitive  statements.  For  the  NIST  laboratories  results  can  be  stated  based 
upon  over  90  years  of  experience. 
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Technology  Impact  Measurement  Issues 

There  is  a  need  to  fight  the  notion  that  you  can  come  up  with  a  simple, 
meaningful,  single,  quantitative  result  for  something  as  complex  as  the  technology 
system.  While  the  political  system  demands  sound  bites,  there  are  none  that  will  answer 
the  technology  questions  adequately.  There  is  a  need  to  fight  oversimplification.  The 
issue  then  becomes:  How  far  back  can  one  provide  a  family  of  quantitative  answers  that, 
in  some  fashion,  one  can  discuss  results  without  using  vignettes.  The  answer  appears  to 
be  that  there  is  a  need  to  convey  the  order  of  magnitude  effects  which  result  from 
technology  investments,  without  being  able  to  specify  the  precise  rates  of  return. 

What  are  good  measures?  Following  are  six  rules  for  ex  post  assessments  and 
measurements: 

•  Measurement  must  be  simple  to  understand  and  relate  to.  A  14th  order  or  a 
15th  order  system  that  has  a  lot  of  inputs  that  are  extremely  time  intensive  to 
develop,  is  a  system  that  is  too  complex.  Furthermore,  industry  is  not  willing 
to  divulge  every  detail  of  their  operations,  so  there  must  be  simplifications 
which  are  realistic  in  this  regard.  Simplicity  also  means  that  measures  must 
have  a  clear  definition  that  is  meaningful  to  the  public.  For  example,  jobs, 
economic  benefits,  and  the  like. 

•  Measurements  must  be  acceptable  to  the  field  of  economics.  As  soon  as  one 
proffers  a  measure  the  economics  profession  will  place  it  under  a 
microscope.  The  result  may  be  assertions  that  the  measure  is  somehow 
incomplete,  inappropriate,  or  simply  incorrect.  Therefore,  measures  must  be 
defensible  in  the  mainstream  fields  of  economics  or  economic  thought. 

•  Measurements  must  be  quantitative  to  the  first  order.  It  is  better  to  get  a 
timely  answer  that  is  ten  to  20  percent  accurate  rather  than  an  untimely  that  is 
one  percent  accurate.  It  is  better  to  describe  how,  based  on  the  $70  billion 
that  the  government  spends,  that  there  are  answers  of  the  same  order  of 
magnitude  in  terms  of  outcomes  or  results. 

•  Measurements  are  always  “needed  yesterday.”  Getting  an  answer  or  set  of 
metrics  down  the  road  does  little  good  when  trying  to  respond  to  issues 
raised  by  the  Congress  within  a  two,  three,  or  four  month  time  frame. 

•  Measurement  must  be  able  to  deal  with  the  panoply  of  the  different  types  of 
government  investments  that  go  on  that  make  up  the  fabric  of  TRP,  ATP, 
MET,  and  the  other  government  R&D  activities. 

•  Measurement  must  include  considerations  of  in-kind  contributions  as  well  as 
facilities  and  resources  that  are  brought  to  the  table  or  shared  by  government 
and  industry  or  within  partnerships.  This  must  also  include  technical 
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expertise,  such  as  specialized  engineering  which  may  be  found  within 
organizations  and  companies. 

SBIR  is  an  example  of  a  technology  development  program  with  some  immediately 
observable  metrics.  What  are  the  economic  benefits  from  the  SBIR  program  outside  of 
its  direct  benefits  to  government  agencies?  There  are  a  number  of  ways  to  make  such 
measurements.  For  instance,  measures  of  the  amount  of  private  capital  investment  that 
results  from  the  government  SBIR  program. 

International  benchmarking  would  be  useful  for  the  NIST  programs.  It  may  be 
wise  to  do  some  international  comparisons  for  programs.  There  are  comparable 
programs  to  each  of  NIST’s  major  elements  in  Europe  and  Asia,  and  one  can  follow  the 
experiences  there.  There  are  evaluations  ongoing  in  Europe  and  in  Japan  which  may  be 
of  use  in  justifying  programs  here.  What  would  be  helpful  to  demonstrate  is  that  one  is 
providing  as  good  a  service  to  industry  or  to  society  as  our  competitors  are  providing. 

The  experience  of  foreign  programs  may  be  able  to  teach  us  much  about  U.S. 
efforts.  Foreign  programs  have  not  attempted  quantitative  estimates  of  the  results  of  their 
investments.  Rather,  they  get  together  panels  of  experts,  and  make  judgments.  It  is  very 
interesting  to  see  that  some  of  the  same  problems  have  arisen  in  terms  of  reviews  in  Japan 
and  in  Europe,  that  the  U.S.  has  had  to  deal  with.  As  in  the  U.S.,  justification  for  foreign 
programs  has  meant  recasting  them  to  explain  them  to  various  constituencies.  This  has 
included  putting  new  names  on  them,  trying  to  “bum  their  trails,”  rather  than  justification 
on  economic  bases.  Today  there  is  agreement  on  the  value  of  more  systemic  and  long 
term  evaluations.  There  was  too  great  a  pressure  on  them  to  show  short  term  results. 
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Session  Three 

METRICS  FOR  MULTIPLE  POLICY  OBJECTIVES: 
THE  CASE  OF  TRP  AND  DUAL-USE 


The  Technology  Reinvestment  Project  (TRP)  is  unique  in  a  number  of  ways.  It 
involves  six  agencies:  the  Departments  of  Defense,  Commerce,  Energy,  and 
Transportation,  NASA,  and  the  NSF.  It  is  intended  to  pursue  commercial/military 
integration,  primarily  through  leveraging  commercial  technologies  for  military  purposes. 
It  seeks  affordable,  dual-use  solutions — a  dual-use  solution  being  one  that  is 
commercially  viable  and  militarily  useful. 

OVERVIEW  OF  THE  TRP 

While  there  have  been  a  number  of  different  public  faces  put  on  the  TRP,  there 
has  been  consistency  in  the  program.  The  original  mission  statement  was:  “To  stimulate 
the  transition  to  a  growing,  integrated  national  industrial  capability  which  provides  the 
most  advanced  affordable  military  systems  and  the  most  competitive  commercial 
products.”  The  key  component  of  the  program  was  to  stimulate  the  integration  of 
military  and  commercial  research  and  production  activities.  This  mission  statement  was 
put  together  in  December,  1992,  and  it  has  survived,  until  today,  internally  in  the  TRP. 

In  designing  the  TRP,  three  different  types  of  goals  were  originally  envisioned. 
The  first  was  economic  impact,  and  some  held  that  this  was  the  primary  objective  of  the 
program.  The  second  view,  primarily  held  by  the  architects  of  the  original  legislation, 
included  converting  defense  industries  to  either  dual-use  or  commercial  capabilities.  The 
third  view,  which  is  the  one  used  in  constructing  the  TRP,  stemmed  from  the  vision  that 
Bill  Perry,  John  Deustch,  and  others,  had  for  “commercial-military  integration.”  This 
latter  view  was  one  which  centered  primarily  on  affordability — the  idea  that  the 
Department  of  Defense  needed  to  expand  its  supplier  base  to  include  not  only  traditional 
defense  industries,  but  also  to  include  commercial  or  commercially  derived  products,  thus 
engaging  the  power  of  the  commercial  sector  of  the  U.S.  in  national  security.  Dual-use 
concepts  foster  a  mid-term  view  of  integration;  that  is,  trying  to  create  today  the 
technologies  that  the  military  can  use  when  it  matures  in  3  to  5  years. 
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TRP  does  not  consider  itself  as  a  defense  conversion  program,  although  some 
observers  have  labeled  it  as  such.  It  is  really  a  defense  transition  program  attempting  to 
move  towards  an  integrated  commercial  and  military  industrial  base.  Conversion 
connotes  something  more  akin  to  what  is  happening  in  the  USSR — ^that  is,  how  do  you 
take  the  capital  base  and  turn  it  from  a  military  capital  base  into  an  industrial  capital  base 
with  commercial  capabilities.  That  is  not  what  the  TRP  is  doing.  There  is  not  enough 
money  to  affect  such  a  change  on  the  part  of  the  government.  The  idea  is  that,  by 
investing  in  dual-use,  technologies  can  be  made  both  militarily  useful  and  commercially 
viable,  so  that  the  DoD  and  the  industrial  world  benefit  together. 

In  addition,  the  strategy  of  commercial-military  integration  envisioned  in  the  TRP 
is  not  compatible  with  the  other  two  views  of  converting  defense  industries.  Conversion 
of  defense  industries  to  commercial  pursuits  is  an  industry  decision,  not  a  government 
decision.  If  companies  want  to  make  a  move  in  that  direction  and  the  TRP  can  help  make 
that  move,  that  is  certainly  a  valid  thing  to  do,  but  it  is  not  the  primary  mission  of  the 
TRP.  Nor  is  economic  impact  a  primary  mission,  but  rather  a  by-product  of  the  program 
and  its  focus  on  dual-use. 

Areas  are  selected  for  funding  under  the  TRP  based  on  military  needs.  The 
second  part  is  to  raise  the  needs  with  industry  to  sort  through  their  views.  A  third  basis 
for  project  selection,  which  has  been  criticized,  was  to  recognize  that  dual-use  might 
include  not  only  military  but  other  government  technology  needs. 

An  ad  hoc  committee  representing  all  of  the  government  partners  in  the  TRP,  the 
six  agencies,  and  now  the  Services  as  well,  meet  to  discuss  the  best  way  to  implement  the 
program.  TRP  gives  those  agencies  the  opportunity  to  suggest  focus  areas.  As  would  be 
expected,  many  of  the  suggestions  received  from  the  other  agencies  tend  not  to  have  a 
clear,  compelling  defense  need.  There  have  been  controversies  because  many  of  those 
inputs  were  not  accepted  because  the  defense  part  of  the  equation  was  not  present. 

During  the  first  round  of  project  selection  there  were  150  need-based  topics 
suggested.  The  group  went  through  a  peer  review  process  to  winnow  that  down  to  a 
manageable  number — originally  12  were  selected.  It  was  the  requirement  for  a  strong 
justification  of  military  need  that  sorted  out  the  project  areas.  The  TRP  then  chose  its 
projects  through  a  free  and  open  competition. 

The  one  thing  TRP  failed  to  do  was  have  title  designations  for  programs.  For 
example,  one  that  is  called  the  “BART  Train”  has  been  severely  misrepresented  in  the 
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press  and  on  the  Hill.  In  fact,  on  that  project  the  aerospace  contractor  has  very  clear, 
militarily  relevant  objectives.  Today,  the  Army  spends  approximately  $60,000  for 
precision  location  equipment  on  various  vehicles.  There  are  two  ways  to  get  that  cost 
down.  One  is  to  redesign  it,  using  leading  edge  commercial  technology.  The  second  is  to 
have  commercial  spin-offs  that  create  economies  of  scale.  That  is  exactly  what  the 
program  is  doing,  and  one  of  the  test  beds  that  the  equipment  will  be  tested  on  are  BART 
Trains  in  the  Bay  area.  Unfortunately,  the  title  of  the  program  says  “BART  Trains.” 
TRP  has  therefore  taken  a  thrashing  by  those  who  want  to  criticize  the  program. 

TRP  is  based  upon  the  idea  that  DoD  can  benefit  by  leveraging  industrial 
resources,  both  in  terms  of  extending  its  own  financial  resources,  and  from  rapidly 
accelerating  commercial  technologies.  It  is  a  50/50  cost  share  program,  as  is  the  ATP 
and  some  other  DoD  programs. 

TRP  is  attempting  to  implement  what  are  called  “other  transactions.”  The  notion 
is  that  the  current  DoD  procurement  process  is  a  real  impediment  to  government  business 
with  the  private  sector.  TRP  attempts  to  overcome  this  by  engaging  in  agreements  which 
are  termed  “other  transactions.”  These  agreements  are  very  much  simplified  procurement 
documents.  They,  allow  companies  and  the  government  to  enter  into  agreements  that  are 
very  similar  to  those  which  would  be  entered  into  among  firms  in  the  private  sector. 

Metrics  to  measure  the  success  of  the  TRP  have  been  an  issue  which  has  dogged 
the  program  ever  since  its  inception.  Congress  wrote  into  legislation  in  TRP’s  second 
year  that  the  program  should  discuss  the  number  of  jobs  created,  the  profitability  of  firms, 
and  other  such  macroeconomics  indicators,  even  before  the  first  contracts  were  let.  This 
has  been  a  real  problem.  In  the  TRP  program  there  is  a  very  simple  rule  for  metrics. 
Good  metrics  have  to  relate  to  a  set  of  original  objectives. 

Industrial-base  impacts  for  dual-use  require  very  different  sorts  of  metrics,  for 
instance,  the  extent  and  success  of  integration  efforts,  the  improvement  of  defense 
affordability,  and  conunercial  spill-over  benefits.  One  way  of  looking  at  this  latter 
impact  measurement  might  be  industrial  based  broadening  and  strengthening. 

Since  development  projects  are  heterogeneous,  they  must  be  handled  on  a  case  by 
case  basis.  In  order  to  do  that,  TRP  is  going  to  be  entering  into  a  series  of  case  studies,  as 
well  as  convening  what  is  being  called  a  “military  and  commercial  evaluation  panel.” 
The  notion  is  that  peer  review  is  required,  both  for  the  investment  portfolio  as  it  is 
proposed,  as  well  as  for  the  status  of  the  ongoing  projects. 
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In  order  to  measure  impacts  and  to  tell  the  TRP  story,  there  is  also  a  need  to 
understand  that  certain  impact  measurements  are  available  at  certain  times.  It  is  not 
possible  to  know  how  many  jobs  are  going  to  be  created  early  on  in  a  program.  It  is  also 
not  possible  to  know  what  the  profitability  of  the  firms  is  going  to  be — even  most 
industry  executives  don’t  really  believe  company  projections  out  5  years.  Therefore, 
while  in  the  TRP  there  is  50/50  cost  sharing  and  companies  claim  that  they  will  be  able  to 
transition  a  product  or  process  into  commercial  use  and  make  a  profit,  it  is  far  from 
certain  what  will  happen.  In  fact,  that  is  part  of  the  risk  undertaken  by  government.  If 
there  is  no  risk,  there  is  no  need  for  government  support. 

There  are  two  kinds  of  risks  in  TRP  projects.  One  is  the  technical  risk,  which  can 
be  reasonably  high.  The  other  part  is  the  economic  risk.  Even  if  you  get  a  product  to 
market,  opportunities  may  vanish  in  following  years.  There  is  a  need  to  understand  that 
success  is  not  simply  getting  something  to  market,  but  that  there  are  long-term  impacts 
and  implications  from  these  technology  investments. 

What  does  an  economist  look  for  to  understand  the  extent  of  success  of 
commercial-military  integration  efforts? 

Evidence  of  co-production.  Production  of  military  and  commercial  items,  either 
on  the  same  production  line  or  in  the  same  facility  using  shared  resources.  In  the  short 
term  one  can  speak  with  companies  about  their  planning  and  their  strategies  for 
penetrating  both  military  and  commercial  markets  and  try  to  understand  whether  they 
intend  to  set  up  separate  facilities  to  produce  the  items  or  whether  they  are  going  to  co¬ 
produce  them.  In  the  long  term  the  sales  of  comparable  products  and  processes  in 
commercial  and  military  markets  is  a  metric  for  integration. 

In  fact,  commercial-military  integration  embraces  a  spectrum  from  commercial 
off-the-shelf,  through  military  unique  products  and  processes,  and  everything  in  between. 
So  we  should  not  expect  to  see  simply  commercial  items  being  used  by  the  military.  We 
may  have  to  paint  them  green,  and  we  may  have  to  change  some  of  the  microcircuits,  or 
we  may  have  to  install  them  differently.  Nevertheless,  with  a  good  understanding  of 
what  to  look  for,  one  can  use  data  such  as  sales  in  both  the  commercial  and  military 
markets,  or  look  at  the  point  of  origin,  or  look  at  the  supplier  base,  and  trace  integration 
back  to  its  source  of  origin. 

Improved  defense  affordability.  It  is  certainly  possible  to  identify  the  costs  of 
military  and  commercial  components  as  they  exist  in  weapons  systems.  This  may  be 
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done  in  the  planning  stage.  Or  one  might  do  this  in  the  production  stage.  Both  are  very 
viable  metrics  and  support  the  claim  that:  “We  planned  to  do  this.”  Even  if,  in  the  long 
term  there  are  changes  in  plans,  one  at  least  has  a  way  of  pinning  down  facts  and  stating, 
specifically,  “This  is  what  I  am  going  to  try  to  do;  this  is  what  we  think  the  impact  is 
going  to  be.”  Or,  “We  are  producing  it  this  way,  and  this  is  what  the  level  of  integration 
is.” 

Industrial  base  broadening  and  strengthening.  This  has  also  been  termed 
“preservation  of  critical  defense  industrial  base  capabilities.”  In  this  case  one  would  look 
at  whether  or  not  TRP  investments  led  to  the  continuation  of  on-shore  production  of  a 
militarily  important  item.  In  some  cases  this  would  be  associated  with  the  need  to  retain 
control  of  production  for  items  which  have  particularly  long  production  lead  times.  One 
may  also  look  at  long  term  military  and  commercial  contractual  relationships  as  the  TRP 
has  attempted  to  get  defense  firms  and  commercial  firms  to  work  together  and  come  up 
with  new  products  and  processes  that  have  dual  uses. 

Industrial  base  broadening  and  strengthening  through  integration  also  must  deal 
with  the  different  business  cultures  of  defense  and  commercial  firms.  These  cultural 
differences  are  enormous.  By  having  commercial  and  defense  firms  work  together  on 
TRP  projects,  we  are  creating  a  “human”  bridge,  a  knowledge  bridge,  an  experience 
bridge  between  firms  in  the  two  sectors.  This,  of  course,  leads  to  technology  transfer 
among  the  team  members. 

ROUND  TABLE  DISCUSSION 

Technology  Reinvestment  Project  (TRP) 

What  are  appropriate  indicators  that  a  spin-off  or  spin-on  project  will  have  the 
desired  affordability  characteristics?  The  metrics  for  initiating  and  monitoring  commer¬ 
cial  projects  in  the  private  sector,  and  tracking  them  after  the  fact,  are  very  well 
established  in  industry.  The  issue  here  is  the  idea  that  military  need  can  spin-off  a 
commercial  product  that  will  reduce  the  cost  of  the  military  product.  Presumably,  if  that 
reduction  is  going  to  be  substantial,  then  the  commercial  marketplace  has  to  become  the 
dominant  user.  The  main  objective  of  TRP  is  to  take  advantage  of  products  that  are 
already  predominant  in  the  commercial  marketplace.  The  case  of  semiconductors  would 
be  a  very  good  example,  as  would  the  case  of  flat  panel  displays. 
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Why  are  TRP  and  Title  III  necessary  to  insert  new  technologies  into  military 
systems?  The  TRP  and  Title  III,  in  some  cases,  are  being  used  in  lieu  of  technology 
insertion  budgets  for  the  Services.  This  is  because  the  Services  do  not  have  insertion 
budgets.  Nor  do  they  have  some  of  the  mechanisms  allowed  the  TRP  to  move  some  of 
these  technologies  forward  through  the  use  of  other  agreements. 

One  of  the  purposes  of  a  TRP  metric  would  be  to  identify  how  broadly  a  dual-use 
approach  could  spread  inside  of  military  operations.  There  is  both  the  question  of  spin¬ 
off,  which  may  be  easier,  and  spin-on,  where  many  other  factors  must  be  considered, 
including  military  suitability.  Under  what  circumstances  does  this  work,  and  what  are  the 
obstacles  to  doing  it?  Many  civilian  technologies  are,  by  some  measures  of  performance, 
way  ahead  of  the  military.  We  also  know  that  the  military’s  needs  are  quite  different  so 
that  it  is  not  so  easy  to  go  from  highly  sophisticated  civilian  components  directly  into 
military  applications.  Commercial  firms  would  go  through  a  series  of  steps  and  not  make 
a  large  jump. 

Obviously  today  the  only  metrics  you  have  for  TRP  are  performance  of  the  TRP 
project  because  they  are  at  most  2  years  old.  Most  TRP  projects  have  an  average  age  of 
a  year  or  a  year  and  a  half.  So  the  only  metrics  you  have  today  to  look  at  are:  “How  are 
those  programs  performing  against  goals  that  were  established?”  That  includes — and, 
this  is  what  is  a  little  different  about  TRP  or  dual-use — an  assessment  of  the 
commitments  made  to  commercialization. 

Where  is  the  team  or  company  involved  on  the  path  to  sorting  out  commercial 
markets?  That  is  not  simply  a  quantitative  assessment.  In  addition  you  have  the  fairly 
standard  cost  schedule  performance  metric. 

Also,  you  have  a  metric  with  respect  to  who  the  players  are.  If  one  reviews  the 
Service  R&D  programs,  how  many  of  them  involve  defense  firms,  and  how  many 
involve  both  commercial  and  defense  firms  (dual-use)  you  will  find  some  major 
differences  when  compared  with  the  TRP. 

The  long  term  for  TRP  projects  is  3  to  5  years.  At  that  point  one  can  begin  to  use 
conventional  commercial  metrics  that  are  well  known  and  well  established.  One  can  also 
use  conventional  military  metrics  forjudging  the  impact  of  specific  TRP  projects.  Those 
are  fairly  well  known. 

Actions  which  enable  integration  can  also  be  measured,  for  example  the  reduction 
of  military  standards  and  military  specifications.  It  is  possible  to  measure  the  degree  to 
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which  contracts  have  changed,  solicitations  have  changed,  and  the  degree  to  which 
military  specifications  are  listed  in  documents. 

A  crossover  point  for  short  term  versus  long  term  metrics  for  dual-use  technology 
would  be  achievement  of  joint  production  which  should  lead  to  economies  of  scale  and 
also  economies  of  scope  because  there  are  different  performance  requirements  for 
defense  versus  the  civilian  market. 

In  defense  we  measure  products  in  three  ways:  cost,  schedule,  and  performance. 
We  should  be  able  to  take  each  of  those  three  measures  and  ask  the  impact  of  the  TRP 
program:  What  was  the  cost  reduction  per  unit?  Did  we  get  it  sooner  than  we  otherwise 
would  have  gotten  it?  Do  we  consider  that  worthwhile?”  These,  along  with  performance 
measures,  tend  to  be  qualitative  judgments. 

If  TRP  had  not  funded  EPIARS,  would  the  Army  still  have  funded  it?  The  Army 
funded  EPLARS  well  before  TRP  was  ever  formed.  They  abandoned  it  because  it  was 
too  expensive.  TRP  picked  it  up  to  make  it  less  expensive.  The  Army  has  now  stated 
that  they  will  purchase  EPLARS  if  the  TRP  meets  its  cost  reduction  goals. 

What  is  industry’s  view  of  the  TRP.  With  all  of  the  controversy  over  the  TRP 
program,  there  is  certainly  a  divergence  of  opinion.  The  TRP  program  allows  companies 
to  do  some  things  that  they,  perhaps,  would  not  have  otherwise  done.  TRP  makes  the 
decision  to  undertake  some  company  investments  much  easier,  although  it  is  impossible 
to  say  whether  these  investments  would  have  been  undertaken  anyway.  Most  examples 
of  successful  TRP  projects  are  highly  militarily  oriented  and  may  have  industrial  spin¬ 
offs.  For  a  defense  contractor,  it  is  much  easier  to  justify  investments  in  projects  which 
have  a  defense  relevance  and  the  potential  for  a  commercial  spin-off,  than  for  a 
commercial  contractor  where  commercial  markets  are  the  primary  target. 

For  instance,  in  medical  imaging  systems  Loral  had  a  program  prior  to  the  TRP. 
But  the  TRP  program  allowed  Loral  to  take  a  basic  imaging  system,  where  one  takes  all 
images  from  hospitals,  whether  they  be  X-rays,  CAT-scans,  or  nuclear  scans,  and  put 
them  in  digital  form.  This  allowed  Loral  to  leap-frog  by  taking  the  technology  and 
applying  it  to  a  telemedicine.  For  the  military,  the  technology  supports  the  extension  of 
highly  trained  medical  expertise  to  the  hinterlands  or  the  battlefield. 

Loral  also  has  a  TRP  program  in  uncooled  electro-optical  technologies.  The  end 
goal  of  this  project  is  to  bring  the  cost  of  infrared  focal  planes  down  considerably  by  get¬ 
ting  rid  of  the  cooling  apparatus,  which  is  one  of  the  big  cost  drivers  in  the  system  today. 
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Loral  had  an  agreement  with  Honeywell  at  the  time  it  bought  its  IR  imaging  system 
division  to  tap  into  that  technology,  but  there  was  a  big  cost  of  technology  transfer;  and 
there  was  a  need  to  make  some  changes  to  the  teehnical  process.  Again,  the  TRP 
program  made  it  easier  for  Loral  to  do  this.  The  project  is  now  well  underway  and  would 
appear  to  be  extremely  successful  with  a  tremendous  potential  for  the  commercial  world. 

TRW  has  a  program  in  precision  laser  machining  derived  from  its  development  of 
solid  state  lasers,  primarily  on  behalf  of  the  military.  TRP  gave  TRW  the  opportunity  to 
do  initial  development  work  to  demonstrate  commercial  applicability.  It  has  been  an 
exeellent  program,  and  it  has  benefit  on  both  the  military  side  and  the  commereial  side. 
TRW,  being  a  company  with  both  sides  of  that  business  within  the  eorporation,  has  also 
been  helped  by  TRP  moneys  to  bridge  the  corporate  cultural  chasm.  In  pursuing  its  TRP 
project,  TRW  has  attempted  to  quantify  the  share  of  the  marketplace  it  expects  to  gain, 
how  attractive  this  is  going  to  be  in  terms  of  returns,  and  so  forth.  There  is  the  constant 
need  to  update  and  revise  such  assessments,  because  the  world  changes,  but  they  serve  as 
a  guide  for  understanding  what  may  happen  in  the  future. 

In  terms  of  global  competitiveness,  in  the  laser  machine  tool  market,  TRW  is 
quite  concerned  about  losing  market  share  to  the  Japanese  and  Europeans.  Unless  TRW 
brings  products  like  this  onto  the  market,  it  will  lose  global  share.  TRW  has  also 
assessed  the  benefits  of  laser  machining  for  the  environment — a  social  benefit  which 
derives  from  the  reduction  in  automobile  weight  which  becomes  possible,  and  the 
gasoline  savings  which  therefore  results. 

Is  the  controversy  over  the  TRP  a  “red  herring?”  There  is,  obviously,  a  huge 
controversy  over  the  Technology  Reinvestment  Project,  politically.  There  are  questions 
as  to  whether  it  is  a  proper  use  of  government  funds.  There  are  questions  as  to  whether  it 
is  the  right  size.  There  are  the  questions  as  to  whether  it  is  the  proper  use  of  DoD  money. 

All  of  these  are  red  herrings  and  miss  the  fundamental  point,  which  is  that  dual- 
use  is  absolutely  essential  to  the  Department  of  Defense,  that  commercial/military 
integration  is  essential.  Therefore,  the  question  here  is  how  do  you  measure  the  success 
or  failure  of  this  particular  program?  Is  it  a  stalking  horse  for  how  you  decide  means  and 
metrics  for  the  broader  task? 

On  the  defense  side,  since  the  primary  goal  is  affordability,  there  is  the  metric  of 
cost  reduction  against  a  base,  which  offers  some  intermediate-term  measures.  This  would 
also  be  reflected  in  how  much  DoD  is  in  fact  modifying  its  own  processes,  specifications. 
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and  standards.  On  the  commercial  side,  it  turns  out  that  there  is  already  a  well  developed 
methodology  for  determining  the  effectiveness  of  a  development  program,  including 
measures  of  market  share,  cost,  adhering  to  milestones,  and  so  on. 

If  we  sat  down  and  designed  the  tests  for  the  TRP,  or  designed  the  tests  for  dual- 
use,  we  would  be  able  to  measure  what  has  occurred,  what  has  not  occurred,  and  what 
will  take  time  to  occur.  When  we  are  talking  about  technology  we  are  talking  about 
development.  Even  more  importantly,  we  are  talking  about  cultural  changes  that  take 
years,  even  decades.  That  is  why  it  is  important  to  have  intermediate  and  long  term 
measures. 

Dual-Use  Investments 

The  case  of  Flat  Panel  Displays.  Congress  mandated  that  DoD  spend  a  certain 
amount  of  money  on  AMLCD,  Active  Matrix  Liquid  Crystal  Display,  which  is  a  little 
more  narrow  than  the  program’s  original  intent.  The  original  notion  of  Congress  was  to 
create  purchase  commitments  or  incentives  to  help  a  particular  vendor  with  their 
business.  One  of  the  things  done  as  part  of  the  National  Flat  Panel  Display  Initiative  was 
to  use  Title  HI  dollars  to  stimulate  demand  by  the  insertion  of  the  panels  into  weapon 
systems. 

The  Flat  Panel  Display  Initiative,  in  turn,  wanted  to  find  out  what  was  needed  to 
help  the  military  in  gaining  access  to  the  use  of  flat  panel  displays.  This  involved 
canvassing  the  management  offices  of  programs  that  were  using  displays  or  thinking  of 
using  displays,  particularly  those  using  CRTs,  to  find  out  what  they  could  use  in  the  way 
of  incentives  in  terms  of  qualification  or  early  purchase  of  displays.  For  the  $20  million 
thus  far  spent  in  this  program,  there  are  over  $200  million  worth  of  displays  going  into 
systems  that  would  have  used  CRTs.  The  offices  either  did  not  have  a  separate  account 
to  fund  new  equipment  qualification,  or  they  had  insufficient  funds  to  purchase  the  flat 
panels  early  in  the  production  phase  (to  help  vendors  and  helps  lower  production  costs). 

There  is  the  question  of  trying  to  get  the  commercial  into  defense  versus  getting 
defense  into  commercial.  One  might  argue  that  the  objective  for  the  Department  of 
Defense  is  to  get  lower  cost  and  higher  performance  weapons  systems,  and  that 
commercial-military  integration  has  been  concluded  to  be  a  means  to  do  this.  Therefore, 
what  one  needs  is  to  have  some  measures  of  means  and  also  some  measures  of  objective 
in  the  sense  that  the  end  result  is  lower  cost  weapon  systems. 
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In  terms  of  being  able  to  measure  whether  we  have  the  same  product  built 
according  to  military  and  commercial  specifications  with  the  same  performance,  this  is 
difficult  because  the  military  almost  never  builds  a  military  and  a  commercial  variant  of 
a  weapon  system.  There  is  very  good  empirical  data  that  shows  that,  as  weapon  system 
performance  keeps  going  up,  the  cost  keeps  going  up  with  it  in  the  defense  world.  In  the 
commercial  world  as  performance  goes  up,  the  costs  keep  coming  down.  So  there  is, 
probably,  something  that  says  that  we  are  not  violating  a  law  of  nature  by  continuing  to 
drive  costs  down. 

The  Services  probably  would  not  have  embraced  the  concept  of  commercial- 
military  integration  on  their  own  without  some  other  pressure.  The  outside  pressure  is 
the  pressure  on  Service  budgets.  The  reality  is  that  the  force  structure  is  going  to  be 
smaller.  You  have  to  maintain  a  qualitatively  superior  force  at  the  same  time  you  have  to 
buy  sufficient  numbers  of  weapon  systems.  The  numbers  do  not  work  out  unless  you  can 
leverage  the  commercial  sector  to  the  maximum  extent.  As  for  commercial 
specifications,  it  does  not  matter  if  your  PC  takes  five  minutes  to  initiate,  but  if  one  has 
an  incoming  missile,  this  becomes  important.  So  time  to  an  officer  in  the  military  is  a  lot 
different  than  time  to  an  economist. 

The  prime  reason  that  the  commercial  product  is  lower  in  cost  is  volume,  volume, 
and  volume.  That  says  that  you  are  going  to  have  to  look  for  a  place  where  the 
commercial  use  will  outweigh  in  numbers  the  military  use  by  a  factor  of  100  or  a  1,000. 
Can  you  find  a  product  where  the  commercial  use  is  100,000  times  the  military?  If  you 
can  do  that  you  can  guarantee  a  cost  reduction. 

If  you  are  going  to  use  the  commercial  industrial  base,  then  the  products  must  also 
be  successful  in  the  world  marketplace,  and  they  have  to  be  managed  by  industry  and  not 
by  the  military.  That  is  a  very  tough  decision  DoD  must  face  up  to  because,  once  you 
start  imposing  military  requirements  and  timing  on  the  commercial  product,  you  will  kill 
the  commercial  product. 

Defense  has  its  unique  industrial  base  that  has  been  very  content  to  provide 
unique  systems  in  the  past,  because  every  time  you  build  a  new  submarine,  you  build  a 
whole  new  combat  system,  and  it  was  a  nice  couple  of  million  dollars  in  somebody’s 
pocket. 

What  is  the  time  horizon  for  applying  dual-use  metrics?  One  way  of  addressing 
the  time  question  is  to  look  at  the  product’s  life  cycle.  It  may  shock  people  from  the 


64 


commercial  world,  but  the  average  defense  weapons  system  now  takes  16  to  18  years  and 
is  stretching  towards  20  years  from  initial  concept  to  first  production.  One  of  the  motives 
of  commercial-military  integration  is  so  the  defense  can  be  first  to  market  with  the  next 
generation  military  weapons  systems.  To  get  within  commercial  time  horizons  requires  a 
dramatic  change  in  the  whole  acquisition  and  development  cycle.  A  reasonable  time 
might  be  about  5  to  7  years.  But  the  point  is  that  you  are  not  talking  months,  and  you  are, 
probably,  not  talking  even  a  few  years  if  you  are  going  to  look  at  the  ultimate 
effectiveness  of  the  change.  You  can’t  afford  to  wait  that  long  on  individual  projects. 

Technical  and  cost  metrics  are  relevant  to  military  missions.  You  can  come  up 
with  a  number  of  technical  metrics,  you  can  come  up  with  some  cost  metrics,  but  the  only 
way  they  make  sense  is  to  put  them  in  the  context  of  the  systems  that  the  military  is  able 
to  procure.  You  take  the  performance  measures  of  the  systems.  You  can  quantify  those. 
We  do  that  all  the  time,  just  to  justify  them.  But  you  have  to  put  that  in  the  context  where 
DoD  is  spending  40  or  50  percent  less,  and  it  is  buying  systems  that  are  performing  at  the 
same  level. 

Order  of  magnitude  estimates  are  good  enough.  It  is  important  that  one  has  some 
quick  assessment  of  goodness  that  one  can  get  to — ^perhaps  80  percent  good  answers  with 
20  percent  of  the  effort.  Nit-picking  estimates  is  counter  productive  and  ignores  the  order 
of  magnitude  benefits  which  can  result  from  the  pursuit  of  projects.  That  is:  “It  is  good 
enough  to  say  that  you  are  going  to  create  7500  jobs.  Maybe  it  is  6,000.  Why  should  we 
care  as  long  as  the  magnitude  is  correct?” 
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Appendix  A 


ROUND  TABLE  ANNOUNCEMENT 


Assessing  the  Economic  and  National  Security  Benefits 
From  Publicly  Funded  Technology  Investments: 

An  IDA  Round  Table 


On  April  27,  1995,  you  are  invited  to  a 
round  table  discussion  on  the  economic  and 
national  security  dimensions  of  public  sector 
technology  investments  hosted  by  the  Institute 
for  Defense  Analyses  (IDA),  and  jointly 
sponsored  with  the  White  House  Council  of 
Economic  Advisors  (CEA)  and  the  White 
House  Office  of  Science  and  Technology 
Policy  (OSTP).  The  purpose  of  the  round  table 
will  be  to  explore  practical  issues  associated 
with  the  selection  of  technology  investments 
by  various  government  agencies,  and  how  to 
assess  or  measure  their  success.  Our  co-chairs 
will  be  Joseph  Stiglitz,  Member  of  the  CEA, 
John  Gibbons,  Director  of  OSTP,  and 
Christopher  Jehn,  Director  of  the  Strategy, 
Forces,  and  Resources  Division,  IDA.  A 
listing  of  the  distinguished  invitees  from  the 
academic,  business,  and  government 
communities  may  be  found  below. 


Background 

Investments  in  applied  technologies  are  an 
important  part  of  the  current  Administration’s 
efforts  to  foster  long-term  economic  growth 
and  national  security.  Policies  include  explicit 
links  between  federal  government  technology 
programs  and  the  goals  and  priorities  of  U.S. 
industry.  In  this  era  of  highly  constrained 
budgets  the  use  of  technology  funding  to 
further  economic  and  security  goals  involves 
the  following  key  issues: 


•  How  should  agencies  with  applied 
technology  programs  select  where  to 
invest  public  resources? 

•  How  should  we  measure  the  results  of 
our  investments  in  technology 
development  and  application? 

•  What  lessons  have  we  learned  about 
managing  the  intersection  of  economic 
growth,  national  security,  and  the 
pursuit  of  other  national  objectives? 

The  transition  in  federal  science  and 
technology  (S&T)  policy  occurs  against  the 
backdrop  created  by  the  end  of  the  Cold  War 
and  the  simultaneous  intensification  of  global 
economic  competition.  During  the  previous 
half-century,  the  United  States  approached 
S&T  investments  within  a  framework  most 
notably  described  by  Vannevar  Bush — 
government  supported  basic  science  and 
procured  the  first  technological  fruits  of  space- 
and  military-related  research  as  products  for 
NASA  and  the  Department  of  Defense. 
Through  the  mid-1970s,  commercial  spillovers 
from  federally-funded  R&D  and  procurement 
helped  the  United  States  to  maintain  a  clear 
leadership  position  on  all  fronts — apolitical, 
scientific,  military,  and  economic.  The  United 
States  used  its  position  to  promote  open  trade 
and  global  economic  development  to  the 
benefit  of  both  its  allies  and  itself.  In  the 
process,  it  also  assisted  in  the  creation  of 
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sophisticated  economic  competitors,  the  most 
successful  of  whom,  many  believe,  used 
government  policy  more  purposefully  than  the 
U.S.  did  to  leverage  science  and  technology 
for  the  advancement  of  national  welfare. 

For  policy  makers  in  the  United  States,  the 
objective  of  linking  the  complementary 
technology  activities  of  government  and 
industry  has  become  increasingly  important  as 
a  result  of  foreign  competition.  At  the  same 
time,  the  end  of  the  Cold  War  has  profoundly 
altered  the  political  landscape  and  changed  the 
basis  for  government  technology  investments. 
Commercial  spillovers  from  military  research 
have  diminished,  and  today  dual-use  and  spin- 
on  technologies  appear  to  offer  the  greatest 
promise  for  meeting  the  goals  of  national 
security  while  also  stimulating  economic 
growth. 

Current  administration  technology  policy 
emphasizes  the  central  importance  of 
teclmology  investment  in  building  economic 
strength  and  spurring  growth.  This  new  policy 
includes  a  key  role  for  government-industry 
partnerships  in  strengthening  America’s 
industrial  competitiveness.  The  Administra¬ 
tion  also  recognizes  the  need  to  select  these 
programs  carefully  to  ensure  that  they  are 
efficient  and  offer  measurable  results. 

The  strategy  of  cost-shared  government- 
industry  investments  in  high-risk  technologies 
for  commercial  application,  and  dual-use 
technologies  for  application  by  both  military 
and  commercial  industry,  constitutes  a  new 
direction  in  U.S.  security  and  technology 
policy.  Clearly  there  must  be  an  accounting 
for  the  tangible  benefits  the  public  receives 
from  these  taxpayer  investments  and  the 
results  should  be  monitored  and  effectively 
measured. 


The  Round  Table  on  Assessing  the 
Economic  and  National  Security  Benefits  from 
Publicly-Funded  Technology  Investments  will 
review  and  seek  to  improve  the 
Administration’s  ongoing  efforts  to  create  a 
consistent  assessment  framework  for 
measuring  the  results  of  its  technology 
investments.  Invitees  include  many  of  the 
foremost  students  of  the  economics  of 
technological  innovation  from  academia, 
industry,  and  the  federal  government.  We 
hope  to  explore  the  current  “state  of 
understanding’’ — the  applicability  and 
meaning  of  different  types  of  quantitative  and 
qualitative  measures  to  different  types  of 
government  technology  investments,  and  the 
linkages  we  can  identify  between  research, 
innovation,  and  long-term  economic  growth. 

The  Round  Table  is  organized  around  three 
Topic  Areas.  We  have  asked  several  invitees 
to  offer  their  observations  on  each  topic  to 
begin  each  discussion,  and  assigned  facilitators 
to  keep  the  group  focused. 

Topic  Area  Sessions 

I.  Decision  Criteria  for  Technology 
Investments:  Understanding  the 
Rationales  for  Commercial  and 
Dual-Use  Technology  Programs 

The  Round  Table  will  begin  with  a  session 
on  identifying  indicators  which  may  be  used 
to  help  government  decision  makers  target 
their  publicly-financed  investments  on  areas 
likely  to  generate  high  social  rates  of  return. 
Questions  to  be  addressed  include:  What  are 
the  criteria  for  choosing  public  technology 
investments?  How  can  we  identify 
investments  with  a  high  probability  of 
spillovers?  What  can  we  learn  from  the 
experience  of  other  nations,  states,  and  private 


industry  about  identifying  areas  for  investment 
with  a  potential  for  high  social  returns?  And, 
finally,  how  can  performance  measures  be 
used,  along  with  sunset  provisions  and  other 
limits  on  investment,  to  tell  government 
decision  makers  when  to  “pull  the  plug”  on 
public  investments? 

II.  State  of  Understanding  and 
Measuring  Results 

What  do  we  know,  and  when  do  we  know 
it?  What  sorts  of  impacts  are  we  trying  to 
measure?  Are  these  the  right  ones?  The 
emphasis  of  this  session  will  be  to  identify  the 
strengths  and  weaknesses  of  the  available  tools 
for  assessing  public  technology  investments, 
and  how  to  apply  different  sets  of  performance 
metrics  to  different  types  of  technology 
projects?  Questions  addressed  will  include: 
What  are  the  uses  and  limitations  of  the 
quantitative  and  qualitative  measures  we 
currently  employ?  When  are  quantitative 
measures  inappropriate  and  misleading,  and 
how  can  qualitative  approaches  be  made  more 
rigorous  and  broadly  applicable?  Given  that 
the  outcomes  are  only  apparent  in  the  long 
term,  what  sorts  of  short  term  and  interim 
measures  are  appropriate? 

III.  Dual-Use:  Understanding  The 
Intersection  of  Military  and 
Commercial  Benefits 

What  is  dual-use?  What  is  its  role  in 
promoting  a  more  robust  national  security? 
How  can  national  security  benefit  from  dual- 
use  technology  investments  and  an  integrated 
commercial  and  military  industrial  base? 
How  could  such  integration  lead  to  more 
affordable  weapons  systems?  What  do  we 
know  about  managing  the  intersection  of 
defense  needs,  commercial  environments,  and 
economic  growth?  Are  there  lessons  to  be 


learned  from  dual-use  that  will  benefit  public 
technology  investments  for  other  national 
objectives  such  as  health,  energy,  the 
environment,  and  education? 


Appendix  B 


READ-AHEAD  MATERIALS 


Assessing  the  Economic  and  National 
Security  Benefits  from  Publicly-Funded 
Technology  Investments: 

An  IDA  Round  Table 


Read-Ahead  Materials 


April  27, 1995 


Institute  for  Defense  Analyses 
1801  N.  Beauregard  Street 
Alexandria,  Virginia  22311-1772 


Session  1 


SESSION  ONE:  CRITERIA  FOR  SELECTING  AND  EVALUATING 
PUBLIC  SECTOR  TECHNOLOGY  INVESTMENTS 


In  general,  public  investments  in  technology  development  and  diffusion  in  the 
United  States  have  been  aimed  at  achieving  specific  policy  objectives,  such  as  those 
associated  with  national  defense,  environmental  protection,  and  improved  health.  Since 
the  late  1970s,  however,  a  growing  number  of  public  technology  investments  have  also 
been  made  with  an  eye  towards  improving  industrial  or  economic  performance.  Unlike 
their  predecessors,  these  more  recent  investments  have  incorporated  economic 
considerations  as  part  of  their  primary  objectives,  indicating  a  clear  transition  from  a 
neutral  to  a  positive  interest  in  economic  benefits  on  the  part  of  public  decision  makers. 
As  a  result,  decision  makers  must  now  consider  the  following  key  questions:  On  what 
scale  should  economic  benefits  be  measured,  and  what  are  the  most  appropriate  methods 
for  making  such  decisions  when  future  outcomes  are  highly  uncertain? 

The  overriding  economic  rationale  for  public  sector  technology  investments  is 
that  they  may  be  used  to  promote  market  efficiency  when  an  innovation  is  seen  as  easily 
“appropriable.”  By  appropriable  we  mean  a  circumstance  which  arises  when  firms 
believe  that  they  will  be  unable  to  capture  a  sufficient  stream  of  monetary  benefits  from  a 
technology  investment  to  compel  them  to  commit  resources  to  pursue  a  given  innovation. 
As  a  result,  potentially  important  spillovers,  or  “positive  externalities,”  in  the  form  of 
benefits  that  would  become  available  to  other  firms  (or  to  consumers)  will  not 
materialize.  In  such  cases  an  economist  would  state  that  the  social  rate  of  return  to  a 
technology  investment  is  higher  than  the  private  rate  of  return.  In  general  it  is 
recognized  that  when  this  is  true  the  private  marketplace  does  not  lead  to  an  optimum 
social  outcome  since  firms  will  tend  to  underinvest.  As  a  result  government  is  believed 
to  have  an  important  complementary  role  to  play  either  by  sponsoring  research  itself  or 
by  subsidizing  private-sector  research. 

Accepting  the  above  rationale  for  public  sector  technology  investments,  however, 
does  not  relieve  government  decisions  makers  of  the  burden  of  choosing  among  different 
investments  and  accoimting  for  their  ultimate  effects.  Even  though,  in  principle, 
economic  analysis  can  capture  both  the  direct  economic  returns  and  the  “extra”  economic 
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benefits  that  spill  over  to  other  parts  of  the  economy  from  a  publicly  supported 
investment  in  new  or  improved  technology,  the  specific  contributions  of  the  government 
effort  are  exceptionally  difficult  to  quantify. 

As  stewards  of  the  public’s  trust,  government  decision  makers  are  therefore 
confronted  with  two  related  measurement  tasks,  one  involving  the  means  appropriate  for 
choosing  among  proposed  technology  investments,  and  the  other  involving  the  methods 
for  measuring  their  ultimate  results.  In  the  former  case,  the  most  widely  used  approach 
by  government  for  choosing  technology  investments  involves  so-called  peer  review.  In 
the  latter  case,  both  individual  project  case  studies  and  statistical  (econometric)  analyses 
have  been  employed. 

Choosing  Investments  Before  the  Fact:  Peer  Review 

Most  government  investments,  whether  they  are  in  new  technologies  or  direct 
procurements,  are  compelled  by  statute  to  undergo  a  competitive  process  designed  to 
reveal  the  strengths  and  weaknesses  of  proposed  alternatives.  In  the  case  of 
procurements,  when  the  item  to  be  purchased  is  known  or  specified,  financial  costs  are 
key  to  making  decisions.  In  the  case  of  technology  investments,  however,  usually  the 
competition  takes  place  across  heterogeneous  approaches  to  solving  a  problem  or 
advancing  the  frontiers  of  scientific  knowledge  and  technological  know-how.  Also  in  the 
case  of  technology  investments  there  is  generally  extreme  imcertainty  and  therefore  no 
means  of  accurately  making  financial  or  economic  forecasts.  As  a  result,  a  paucity  of 
quantitative  metrics  exists  and  qiialitative  approaches  are  in  widespread  use. 

Typically,  for  the  purpose  of  choosing  among  technology  investments 
government  decision  makers  have  turned  to  some  form  of  peer  review  where  subject 
matter  experts  are  employed  to  advise  the  government  project  managers  on  the  selection 
of  investments.  Of  course  this  approach  is  not  without  its  own  set  of  problems.  For 
instance,  such  reviews  inevitably  rely  a  great  deal  on  the  reputation  and  experience  of  the 
reviewers,  and  the  results  are  often  skewed  by  the  “fraternity”  effect,  wherein  peers  who 
focus  on  very  narrow  areas  of  technology  overestimate  potential  social  returns. 
Conversely,  peers  who  have  built  a  reputation  as  experts  in  particular  technical  areas  may 
also  find  themselves  overestimating  the  risk  of  funding  a  viable  but  unorthodox  approach, 
or  perhaps  they  may  be  unwilling  to  fimd  new,  unknown  investigators. 

Despite  these  weaknesses,  peer  review  will  remain  central  to  the  ex  ante  choice  of 
technology  investments  by  the  public  sector.  We  ask,  should  more  diverse  sets  of 
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reviewers  be  used,  including  experts  from  a  variety  of  science  and  engineering  disciplines 
and  a  variety  of  public  and  private-sector  backgrounds?  Might  it  be  beneficial  to  institute 
a  system  of  checks  and  balances  to  “review  the  reviews?”  If  economic  returns  are  central 
to  the  rationale  behind  technology  investments,  then  should  social  scientists  be  added  to 
review  panels? 

Evaluation  After  the  Fact:  Case  Studies 

Policy  analysts  often  use  case  studies  and  historical  examples  to  assess — ex 
post — ^the  social  returns  from  public  investments  in  technology.  For  instance,  at  the 
microeconomic  level  much  has  been  learned  from  controlled,  structured  comparisons 
between  the  development  of  similar  technologies  in  different  national  contexts.  Such 
evidence  is  used  to  discern  whether  distinct  technology  trajectories  or  “national 
innovation  systems”  exist.  Some  researchers  go  so  far  as  to  infer  that  specific  results  are 
suggestive  about  both  the  possibilities  and  limitations  of  government-industry 
partnerships.  However,  it  is  important  to  not  read  too  much  into  such  anecdotal  evidence. 

In  virtually  every  instance,  technology  developments  singled  out  for  analysis  have 
been  selected  because  they  evidence  particularly  marked  economic  impacts,  positive  or 
negative.  Thus  a  study  confined  to  cases  where  public  technology  investments  appeared 
to  produce  high  social  returns — jet  engines,  computer-controlled  machine  tools,  the 
Internet — ^would  tend  to  reach  conclusions  diametrically  opposed  to  those  of  studies  that 
emphasized  public  technology  investments  that  obviously  failed — synthetic  fuels,  the 
supersonic  transport,  the  fast  breeder  reactor.  Absent  carefully  controlled  comparisons  of 
a  sufficient  number  of  both  ordinary  and  extraordinary  cases,  such  studies  do  not  appear 
to  tell  us  much  about  the  likely  outcomes  from  new  technology  investments,  public  or 
private. 

Evaluation  After  the  Fact:  Statistical  and  Econometric  Analysis 

Attempts  to  quantify  the  economic  impacts  of  public  technology  investments 
through  statistical  (econometric)  analyses  generally  begin  with  a  model  that  attempts  to 
specify  the  relationship  between  the  technological  inputs  (e.g.,  dollar  values  for  R&D 
spending  on  technology  development  or  diffusion)  and  the  economic  outputs  (e.g.,  some 
measure  of  technological  innovation — for  example,  growth  of  output  or  productivity). 
Attempts  have  also  been  made  to  measure  what  is  known  as  “diffusion,”  for  example  the 
number  of  numerically  controlled  machine  tools  adopted  or  the  number  of  production 
workers  trained  to  program  them.  Such  analyses  typically  proceed  in  two  steps.  First, 
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direct  and  indirect  rates  of  return  on  public  investments  are  calculated.  Second,  an 
attempt  is  made  to  determine  whether  the  public  investments  have  promoted  more 
spillover  benefits  than  the  market  would  have  generated  on  its  own,  or  whether  public 
monies  simply  displaced  valuable  private  investment. 

Econometric  methods  tend  to  work  best  for  measuring  the  impact  of  privately 
funded  R&D  because  the  inputs  and  the  outputs  of  R&D  can  be  more  precisely  identified 
for  a  single  firm  or  industry.  In  contrast,  measuring  the  economic  value  of  a  public 
investment  in  technology  requires  a  method  for  addressing  the  indirect  effects  of  the 
public  investment  on  a  vdde  range  of  firms  and  industries.  This  requirement  presents 
conceptual  and  methodological  problems  that  may  limit  the  conclusions  that  can  be 
drawn.  Industrywide  or  economywide  studies  capture  spillovers  to  technology 
investments  more  effectively  than  studies  of  particular  firms  or  sectors,  but  this  presents 
analysts  with  a  choice  between  two  less  than  perfect  options;  narrow  the  investigation  and 
thus  miss  some  of  the  spillovers,  or  aggregate  many  sectors  but  muddy  the  results.  Either 
alternative  limits  the  usefulness  of  the  analysis. 

Measurement  Issues  in  Evaluation 

How  should  we  estimate  the  contribution  from  public  sector  technology 
investments  due  to  a  particular  program?  One  way  is  through  a  process  of  elimination. 
For  instance,  many  investigators  begin  by  modeling  the  quantitative  contributions  of  such 
inputs  as  capital,  labor,  raw  materials,  and  worker  skills  to  outputs,  usually  revenue 
streams  attributable  to  an  innovation.  Ironically,  technological  irmovation  in  such  studies 
shows  up  as  an  “unexplained”  residual — ^it  appears  as  the  unexplained  quantitative  gain  to 
output.  To  derive  the  net  impact  from  public  sector  investments  this  residual  is  then 
compared  with  the  size  of  the  initial  technology  investment.  The  real  complexity  in 
performing  such  calculations  lies  not  in  making  the  calculations,  but  in  obtaining 
believable  data. 

It  is  notoriously  difficult,  for  instance,  to  assign  quantitative  values  to  qualitative 
improvements  brought  about  by  a  new  technology — especially  when  the  technology 
appears  in  the  form  of  an  entirely  new  product  line  or  a  brand  new  industry  created  over  a 
period  of  years.  It  also  turns  out  that  the  benefits  accruing  to  producers  in  the  form  of 
profits  are  easier  to  measure  than  the  benefits  that  accrue  to  consumers  in  the  form  of 
lower  prices  or  the  ability  to  do  new  things.  These  benefits  can  only  be  estimated  using 
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the  concept  of  “consumer’s  surplus,”  that  is,  deriving  a  measure  of  the  premium  that 
consumers  would  have  been  willing  to  pay,  on  average,  for  the  new  product. 

Because  of  the  problems  associated  with  tracing  quantitative  impacts  from 
technological  changes,  studies  of  the  return  on  investment  in  technology  tend  to  show 
higher  impacts  for  investments  resulting  in  process  innovations  (i.e.,  making  existing 
products  more  efficiently)  than  for  those  that  result  in  product  innovations  (i.e.,  creating 
new  types  of  products  or  new  versions  of  existing  ones).  Perhaps  more  important,  studies 
of  incremental  improvements  in  technologies  also  appear  to  offer  more  believable  and 
tractable  estimates  of  the  economic  impacts  from  technological  changes  than  do  studies 
of  the  effects  of  far-reaching,  radical  innovations,  such  as  integrated  circuits.  The 
problem  in  estimating  the  impacts  of  such  truly  revolutionary  technologies  is  that  they 
tend  to  generate  structural  transformations  in  the  economy  and  give  rise  to  entirely  new 
product  families.* 

Hard  as  it  is  to  estimate  precisely  the  output  of  a  public  technology  investment, 
such  measurements  may  be  relatively  straightforward  compared  with  the  measurement  of 
the  investment  itself— the  “input.”  In  other  words,  how  does  the  analyst  characterize  the 
activity  of  a  particular  program  or  public-private  partnership?  Statistical  models  typically 
measure  only  the  amount  of  money  spent  on  the  government-organized  R&D  effort  and 
thus  exclude  quantification  of  subsequent  downstream  activities— essential  inputs  to 
technological  innovation — such  as  “learning  by  doing”  in  production  and  “learning  by 
using”  in  consumption.  In  addition,  value  attributed  to  a  public  investment  in  technology 
development  or  diffusion  must  take  into  account  both  the  time  horizon  for  which  the 
benefit  is  being  calculated  and  the  objective  for  which  the  project  is  being  undertaken, 
e.g.,  fundamental  research,  technology  development  for  public  missions,  pre-commercial 
technology  development,  and  industrial  modernization  or  diffusion. 

Issues  for  Discussion 

•  In  the  areas  of  peer  review  and  evaluation  (quantitative  and  qualitative),  which 

limitations  are  fundamental,  and  which  can  be  overcome  with  continued  effort? 


*  Public  investments  in  technology  that  contribute  more  to  product,  as  opposed  to  process  innovation, 
and  to  radical  innovation,  as  opposed  to  more  incremental  progress,  may  in  fact  be  undervalued  by 
econometric  analyses.  Such  analyses  measure  average  rates  of  remm,  not  marginal  rates  of  remm. 
Since  we  expect  average  remrns  to  exceed  marginal  remrns,  we  know  that  quantitative  smdies  are  apt 
to  overstate  shifts  at  the  margin,  which  is  precisely  where  public  technology  investments  have  their 
effects. 
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•  What  can  we  leam  from  the  experiences  of  companies  and  academic  researchers 
in  the  use  of  these  methods? 

•  What  can  be  learned  from  past  attempts  at  publicly  supported  technology 
development  in  the  United  States  and  contemporary  efforts  in  other  countries? 
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Session  2 


SEHING  PRIORITIES  AND 
MEASURING  RESULTS 


Growing  appreciation  of  technology’s  pivotal  economic  role  is  leading  to 
changes  in  federal  R&D  investment  strategies.  The  Clinton  Administra¬ 
tion  aims  to  accelerate  technology  development  and  ^plication  as  part 
of  a  national  effort  to  foster  long-term  economic  growth  that  creates  new  high- 
quality  jobs,  builds  new  industries,  and  improves  the  U.S.  standard  of  living.  With 
strong  support  in  Congress  and  industry,  the  Administration  advocates  direct,  pur¬ 
poseful  investment  in  commercially  relevant  technologies.  Moreover,  the  new  pol¬ 
icy  assigns  the  federal  government  to  the  role  of  partner  to  industry — ^as  well  as 
to  labor  and  academia — in  woiidng  to  catalyze  and  facilitate  technology  devel¬ 
opment,  ^plication,  and  adoption.  An  operational  aim  is  to  focus  and  levert^e 
federal  etq)enditures  so  that  relatively  small  investments  yield  meaningful 
economic  benefits  for  the  nation. 

Like  the  post-World  War  11  decisions  that  successfully  aligned  federal  R&D  spend¬ 
ing  with  defense  and  other  national  needs,  today’s  policy  decisions  are  reckoning 
with  new  circumstances  wrought  by  the  end  of  the  Cold  War;  intensifying  global 
economic  competition,  the  rapid  diffusion  of  discoveries  and  innovations  across 
international  borders,  and  the  increasing  technological  intensity  of  modem 
manufacturing  and  service  industries. 

This  transition  in  federal  technology  policy  is  occurring  during  a  period  of  con¬ 
strained  federal  budgets,  prompting  two  frequently  asked  questions:  First,  how  do 
agencies  with  technology  programs  identify  and  select  areas  warranting  invest¬ 
ment  of  federal  resources?  Second,  how  do  agencies  measure  the  results  of  their 
technology  investments  and  ascertain  whether  these  investments  are  yielding 
their  anticipated  national  benefits? 

The  Mission  and  Role  of  NIST 

These  questions  are  especially  relevant  to  the  National  Institute  of  Standards  and 
Technology,  part  of  the  Commerce  Department’s  Technology  Administration.  The 
Clinton  Administration  has  assigned  NIST  to  an  innportant  role  in  its  plans  to 
help  U.S.  industry  to  improve  its  development,  commercialization,  and  adoption 
of  new  technology. 

NISTs  explicit  mission  is  to  promote  U.S.  economic  growth  by  woridng  with  in¬ 
dustry  to  develop  and  apply  technology,  measurements,  and  standards.  [See  box 
on  next  page,  NIST  in  the  Context  of  Federal  R&D.] 


An  operational  aim  is 
to  focus  and  leverage 
federal  expenditures 
so  that  relatively 
small  investments 
yield  meaningful 
economic  benefits 
for  the  nation. 
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The  agency’s  portfolio 
of  programs  features 
four  approaches  to 
carrying  out  its 
mission  and  serving 
U.S.  industry. 


Under  this  mission,  unique  among  federal  agencies,  NIST’s  direct  customer  is 
U.S.  industry,  which,  in  turn,  is  the  means  to  accomplishing  the  agency’s  ulti¬ 
mate  objective  of  fostering  sustained  economic  growth  that  benefits  U.S.  citizens. 
The  agency’s  portfolio  of  programs  features  four  approaches  to  canying  out  its 
mission  and  serving  U.S.  industry: 

♦  an  Advanced  Technology  Program  (ATP)  that  invests  in  cost-shared 
projects  in  companies  to  develop  enabling,  high-payoEf  technologies 
that  otherwise  would  not  be  pursued  because  of  technical  risks  and 
other  obstacles  that  discourage  private  investment; 

♦  laboratory  research  programs  focused  on  meeting  U.S.  industry’s  in¬ 
frastructural  technology  needs,  including  standards,  measurements 
and  measurement  technologies,  evaluated  data,  manufacturing 
process  models,  product-performance  tests,  and  quality-assurance 
techniques; 

♦  a  Manufacturing  Extension  Partnership  (MEP)  that  is  scaling  up  to  a 
cost-shared,  integrated,  nationwide  networitof  over  100  manufactur¬ 
ing  extension  centers  to  help  small  and  medium-sized  manufacturers 
to  modernize  their  production  cq)abilities;  and 

♦  the  Malcolm  Baldrige  National  Quality  Award  program  that  provides 
criteria  for  assessing  quality  management  and  competitiveness  and 
sharing  information  on  successful  strategies. 


NIST  in  the  Context  of  fedml  R&D  f  '  i 

The  Clinton  Administration  has  b^;un  to  make  a  dear  transition  in  tedmol- 
ogypolicy.  Itismovii^  tbe  government  fiom  the  four-decade-old  R&D  poIiqr  ’  4 
based  on  post-Worid  Warn  priorities  to  a  policy  that  specifically  indudes  an  ' 


nfss  NIST’s  programs  are  one  inqiottant  part  of  this  trandtion. 


The  federal  government  is  responding  to  global  economic  changes  by: 

♦  Direct  investment  in  dvilian  technology  for  economic  growth. 
NIST  is  the  part  the  federal  R&D  investment  that  focuses 
e}q)lidtly  on  this  mission. 

^  Increased  enq)hasis  on  diiectir^  defense  R&D  toward  a  dual-use 
technology  base.  The  Defense  Department’s  Advanced  Research 
Projects  Agency  has  led  technology  efforts  to  foster  the  growth  of 
an  int^rated  industrial  base  that  is  economically  competitive 
and  able  to  meet  military  needs.  This  approach  is  now  expanding 
to  other  parts  of  the  defense  investment 
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NIST  aims  to  be  a  strong  partner  to  U.S.  industry.  Over  nine  decades  of  working 
with  U.S.  companies,  its  laboratories  have  developed  a  culture  of  cooperation. 
Indeed,  a  recent  report  by  the  Council  on  Competitiveness  described  the  NIST 
laboratories’  process  of  working  cooperatively  with  industry  as  “the  most  stream¬ 
lined  of  all  and  is  perhaps  the  best  model  for  other  federal  labs  to  follow. ...  NIST 
is  very  flexible  and  able  to  respond  quickly  to  industry's  inquiries  without  bureau¬ 
cratic  interference.”^  Tliis  traditional  culture  is  the  base  upon  which  NIST  will 
build  to  meet  its  new  challenges. 


“NIST  is  very  flexible 
and  able  to  respond 
quickly  to  industry’s 
inquiries  without 
bureaucratic  interference.” 


Need-Focused  Priorities, 
Results-Driven  Measures 

NIST  programs  are  guided  by  measurement  and  evaluation  systems  that  the 
agency  uses  as  it  sets  priorities,  evaluates  operational  performance,  and  assesses 
near-  and  long-term  returns  on  agency  investments  and  activities.  Priorities  are 
set  and  results  are  measured  on  the  basis  of  benefits  realized  by  U.S.  industry. 

Conferences  and  workshops  regularly  convened  by  NIST  are  among  a  variety  of 
tools  that  the  agency  uses  to  identify  industry’s  high-priority  technology  needs. 
Industry  input  helps  to  set  the  direction  and  emphasis  of  NIST  programs,  and  it 
encour^es  industries  and  individual  companies  to  shq)e  and  participate  in  new 
initiatives.  One  result  of  this  process  is  high  levels  of  industrial  participation  in 
NIST  programs;  another  is  solid  agreement  on  technical  objectives. 


Dqpartineat  c^  Eoergy  and  the  Natirmai  Aeronautics  and  Spa^ 
Adooinistration,  forexanQ)le,  are  wodting 
toiy  investments  for  economic  benefit  , 

♦ 

tinue  to  be  a  critical  source  for  future  technological  progress.  Ibe  { 
National  Science  Foundation  and  the  National  Institutes  of 
Ifealth  are  key  players  in  this  endeavor 

In  the  context  (tf  these  new  directions  in  federal  tedmolr^  policy,  NIST  has 
an  important  tightfy  focused  role  to  play.  As  the  NIST  budget  grows,  the 
agency’^sharer^furuiingvriUincreasebutvriUstiUrernamasmallfiractian 
of  overall  ^pending  less  than  two  percent  of  the  federal  allocation  and 

less  than  one  percent  of  the  nation's  to^  RSD  e]q)enditutes.  Woridng  in  part- 
nership  with  industry  and  coordinating  closely  with  otherfederal  tedmolc^ 
agencies,  NIST  can  omtribute  significantly  to  the  national  effort  to 
accelerate  the  benefits  of  technology  for  economic  growth. 
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NIST  seeks  and  relies  on  industry  input  on  the  direction  and  content  of  its  pro- 
gi^  at  several  levels  of  organization.  For  example,  the  Visiting  Committee  on 
Advanced  Technology,  a  nine-member  advisory  body  that  traces  its  origins  to  the 
founding  of  the  agency  in  1901  and  is  composed  largely  of  industry  repre¬ 
sentatives,  meets  quarterly  to  review  the  policies,  budget,  organization,  and 
programs  of  the  Institute.  Indicative  of  the  committee’s  oversight  are  a  1992 
evaluation  of  NIST’s  strategic  planning  process,  subsequent  reviews  of  the  strate¬ 
gic  plans  of  several  laboratories,  and  a  1993  study  of  the  strategic  direction  of  the 
Advanced  Technology  Program.  In  addition,  each  of  NISTs  eight  intramural 
laboratories  undergoes  annual  performance  assessments  by  independent  panels 
of  experts  from  industry  and  academia,  convened  by  the  National  Research 
Council. 

Continuous  Improvement 

Like  any  organization  aspiring  to  woild-class  status,  NIST  must  critically  evalu¬ 
ate  itself  and  scrutinize  its  programs  through  the  eyes  of  its  customers.  Likewise, 
systems  and  mechanisms  for  setting  priorities,  evaluating  performance,  and 
measuring  impacts  must  be  continually  strengthened  and  improved,  paving  the 
way  for  improvements  in  the  quality  and  content  of  programs  and  services. 


Scaling  Up,  Managing  Growth 

President  Clinton’S  econraiic  plan  calls  for  increasing  NIST^  total  budget  ; 
ftom  $384  million  in  FY 1993  to  $1.4billion  in  FT  1997.  While  the  total  NIST 


The  strategy  is  to  build 
on  NIST’s  proven 
approaches  for  working 
with  industry  and  to 
achieve  scaie-up  with 
minimum  staff  growth  and 
bureaucratic  overhead. 


represents  a  ggnifirgnt  challenge  for  NIST  to  scale  up  rapidly  ^diile  maintain- 
ing  the  quality  of  its  activities. agency  is  meeting  the  challenge  throu^ 
careftilly  d^velopfid  manayment  plans.  The  strategy  is  to  build  on  NISTs  I 

proven 

minimum  staff  growth  and  bureaucrahe  overtiead. 

TWo  programs  will  account  for  more  than  half  of  the  e]q)ected  increase  in 
the  agency’s  budget  through  1997.  The  ATP  will  grow  from  $68  million  in 
FY 1993  to  $750  million  in  FY 1997.  The  MEP  will  increase  from  seven  manu¬ 
facturing  technology  centers  to  a  nationwide  network  of  over  100  extension 

centers.  Both  programs  have  been  pilot  tested,  and  results  attest  to  their  poten¬ 
tial  to  produce  important  macro*  and  microeconomic  benefits.  From  their 
inception,  the  ATP  and  MEP  have  been  designed  to  be  national  in  scope.  And, 
because  the  funds  in  each  program  are  for  activities  performed  in  the  private 
sector,  neither  will  require  lar^  increases  in  NIST  persormel.  For  FY 1997, 
each  program  will  operate  with  a  total  staff  of  under  100  staff  members. 
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The  focus  on  methods  for  setting  priorities,  evaluating  operational  performance, 
and  measuring  results  is  e^ially  critical  now,  as  the  Administration  proceeds 
with  plans  to  nearly  quadruple  the  NIST  budget  by  1997  and  as  public  expecta¬ 
tions  for  economic  returns  on  the  agency’s  technology  programs  increase  com- 
mensurately.  [See  box  on  Scaling  Up,  Managing  Growth.]  The  ATP  and  MEP  will 
scale  up  rapidly,  growing  from  pilot  projects  to  programs  of  the  size  necessary  to 
achieve  national  economic  impact. 

Each  NIST  program  aims  to  accomplish  objectives  that  depend  significantly  on 
industry’s  behavior,  cqiabilities,  and  commitment.  Measurements  of  perform¬ 
ance  and  impact  are  critical,  the  basis  for  improving  the  effectiveness  and  extend¬ 
ing  the  reach  of  NIST  programs.  Yet,  there  is  little  precedent  in  the  federal 
government — and  inconsistent  results  in  industry — ^when  it  comes  to  measur¬ 
ing  the  results  of  technology  investments. 

Returns  on  R&D  investments  typically  do  not  begin  to  accrue  until  several  years 
after  research  is  completed.  This  1^  between  expenditures  and  returns — and 
diffusion  of  expected  benefits  across  broad  sectors  of  the  economy — makes 
assessments  of  the  economic  impact  of  most  ATP  projects  and  much  of  NISTs 
laboratory-based  R&D  a  long-term  endeavor.  Because  economic  effects  can  be 
projected  but  never  known  with  certainty  in  advance,  NIST  must  select  priorities 


Each  NIST  program 
aims  to  accomplish 
objectives  that  depend 
si^ificantiy  on  industry’s 
behavior,  capabilities, 
and  commitment 


Direct  funding  of  NIST  laboratories  will  loi^^  double  between  FY  1^3 
FY 1997,  to  about  $d00  million.  This  increase  will  address  a  shortfall  in  n#f 
tional  exility  that  is  the  result  of  two  trends.  First,  during  the  past  sev^; 


decades,  NISTs  laboratory  budget  has  been  flat  or  eroding  ^e  new  indm^>|^. 
tries — such  as  biotechnology  and  computer  netwoddng — have  emerged  a^^ 
the  technological  complexity  of  establlM  industries  hre  increased.  Thei^"^ 
suit  has  been  a  widening  g^qi  in  our  national  measurement  infrastracture.  7^ 
Second,  NIST  laboratories  have  been  heavily  dqiendent  on  funding  from 
other  government  ^endes:  for  exan^Ie,  about  25  perooit  d  die  NIST  labori^^l 
tory  staff  was  supported  by  Department  of  Defense  funding  in  FY  1993.  As  'I  • ' 

direct  appropriations  for  NIST  laboratories  increase,  the  j^ency  will  be  better 
able  to  respond  to  industry  needs  for  inftastructural  technologies.  The  labora¬ 
tories  plan  to  offset  their  heavy  dependence  on  other  agendas’  funding  and 
thus  only  increase  staff  by  about  10  percent  dirough  FY  1997. 


Ihe  Malcolm  Baldrige  National  Quality  Award  program  already  has  experi¬ 
enced  dramatic  scale-up  from  its  establishment  in  late  1987  by  Congress  to 
its  first  awards  the  next  year  to  a  full-scale  program  with  national  and  inter¬ 
national  irrq)act  With  pressing  needs  to  move  quality  management  prind- 
pies  into  the  practices  of  education  and  healthcare  organizations,  NIST  is 
readying  pilot  efforts  leading  to  full-fledged  award  programs  in  1996. 
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caiefuliy  and  rely  on  measures  of  perfonnance  and  short-  and  intermediate-term 
results  as  it  decides  vrfiere  and  how  to  invest  its  resources. 

In  addition,  methodologies  for  measuring  economic  impact  are  still  evolving, 
The  need  to  measure  and  data-collection  efforts  confront  several  obstacles:  the  proprietary  nature  of 

returns  on  taxpayer  certain  infoimation,  the  necessarily  qualitative  nature  of  some  types  of  impacts 

investments  in  federal  (for  example,  the  adoption  of  quality  strategies  or  changes  in  the  horizons  and 
technology  programs  is  of  composition  of  corporate  research  portfolios),  the  high  cost  of  collecting  original 
paramount  importance,  difficulty  in  clearly  defining  criteria  for  quantifying  job  creation 

and  job  retention.  Methodological  and  data-collection  issues  notwithstanding, 
the  need  to  measure  returns  on  ta3q)ayer  investments  in  federal  technology  pro¬ 
grams  is  of  paramount  inqwrtance,  and  approaches  to  meeting  this  need  warrant 
thoughtful — but  action-oriented — discussion  inside  and  outside  of  government 


ADVANCED  TECHNOLOGY 
PROGRAM 


One  of  the  most  innovative  elements  of  the  NIST  technology  strategy, 
the  ATP  invests  directly  in  the  nation’s  economic  growth  by  supporting 
enabling  technologies  with  strong  potential  for  U.S.  economic  benefit. 

"Hie  ATP  provides  funds  for  the  early  phases  of  technology  development  through 
cooperative  research  jpeements  to  single  businesses  or  industry-led  joint  ven¬ 
tures  and  research  consortia.  There  are  some  legislated  limits.  Awards  to  individ¬ 
ual  companies  may  not  exceed  $2  million,  and  the  projects  must  be  completed 
within  three  years.  Projects  by  joint  ventures  may  run  as  long  as  five  years,  and 
the  ATP  can  fund  up  to  50  percent  of  the  project.  Cost  sharing  is  required  for  all 
projects. 

ATP  projects  are  selected  on  the  basis  of  a  rigorous  competition  that  considers 
both  the  technical  and  business  merits  of  the  proposals.  One  of  the  unique  fea¬ 
tures  of  the  AFP  is  this  review  process,  vdiich  evaluates  the  proposal’s  potential 
economic  impact,  the  evidence  that  the  proposer  is  significantly  committed  to 
commercializing  the  results  of  the  project,  and  other  business-related  factors 
affecting  the  likelihood  that  successful  results  will  be  commercialized. 

The  nature  of  the  ATP  poses  unique  challenges  for  effectively  setting  priorities 
and  evaluating  the  results  of  the  program; 

♦  The  potential  ATP  client  base — effectively  all  of  U.S.  industry — 
huge  and  diverse,  covering  all  areas  of  technology  and  ranging  from 
small  entrepreneurial  start-ups  to  m^or  multinational  corporations. 
'Hie  ATP  relies  on  substantial  input  from  this  diverse  industrial  base 
to  define  and  implement  its  programs.  The  diallenge  is  to  run  a  pro¬ 
gram  that  responds  effectively  to  the  priorities  of  this  heterogeneous 
community. 

♦  Tlie  time  line  for  ATP  payoff  is  relatively  long.  The  ATP  does  not  sup¬ 
port  product  development,  but  rather  the  development  of  key  tech¬ 
nologies  that  enable  innovative  new  or  improved  products,  services, 
and  industrial  processes.  While  the  goal  of  the  AFP  is  to  foster  signifi¬ 
cant  economic  benefits  for  the  United  States,  the  greatest  benefits  will 
flow  from  products,  services,  and  processes  that  will  be  developed 
after  an  AFP  project  itself  is  completed.  The  challenge  is  to  set  up  an 
evaluation  strategy  that  considers  both  the  immediate  performance  of 
the  program  and  its  effectiveness  in  the  long  run. 


The  ATP  relies  on 
substantial  input  from 
this  diverse  industrial 
base  to  define  and 
implement  its  programs. 
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Setting  ATP  Priorities 

A  central  tenet  of  the  ATP  since  its  inception  in  1990  always  has  been  that  its 
research  priorities  should  be  set  by  industry  rather  than  the  government.  The 
purpose  of  the  program  is  not  to  impose  government’s  judgment  of  the  best 
opportunities  for  commercial  success,  but  rather  to  enable  industry  to  pursue 
certain  high-risk  projects  that,  if  successful,  would  enable  significant  economic 
benefits  for  the  country.  U.S.  industry  invests  tens  of  billions  of  dollars  annually  to 
turn  “technology”  into  products,  profits,  and  jobs.  The  relatively  minor  funding 
from  the  ATP  is  meant  to  extend  industry’s  reach,  to  foster  riskier  projects  that 
would  not  be  pursued  by  private  funds  alone. 

During  the  pilot  phase  of  the  ATP,  NIST  held  general  competitions  for  ATP  fund¬ 
ing,  open  to  all  areas  of  technology.  All  project  ideas  come  from  industry.  Candi¬ 
date  projects  are  not  evaluated  on  the  basis  of  what  technology  is  proposed,  but 
on  how  sound  the  proposal  is  within  that  technology  and  on  the  projected  eco¬ 
nomic  impact  Specifically,  proposals  are  evaluated  on: 

♦  scientific  and  technical  merit; 


ATP  focused  programs 
will  have  aggressive, 
well-defined  technological 
and  business  goals, 
generally  involving  the 
parallel  development 
of  a  suite  of  interlocking 
R&D  projects. 


♦  the  potential  broad-based  economic  benefits; 

♦  plans  for  eventual  commercialization  of  the  research; 

♦  the  experience  and  qualifications  of  the  proposer,  and 

♦  evidence  of  the  proposer’s  level  of  commitment  to  the  project  and 
clarity  and  appropriateness  of  the  proposer’s  management  plan. 

As  it  grows  from  a  pilot  program  to  a  full-scale  activity,  the  ATP  requires  a  more 
sophisticated  jqiproach  to  setting  investment  priorities.  Even  at  the  Administra¬ 
tion’s  proposed  funding  level  of  $750  million  per  year  by  1997 ,  the  ATP  will  repre¬ 
sent  less  than  one-half  of  one  percent  of  the  nation’s  R8d)  budget  To  obtain 
rpavimnm  benefit  from  its  investments,  the  ATP  will  devote  the  bulk  of  its  funds 
to  selected,  focused  program  areas. 

ATP  focused  programs  will  have  aggressive,  well-defined  technolo^cal  and 
business  goals,  generally  involving  the  parallel  development  of  a  suite  of  inter¬ 
locking  R&D  projects.  By  mani^g  groups  of  projects  that  will  complement  and 
reinforce  each  other,  the  ATP  will  be  able  to  have  the  greatest  possible  impact  on 
technology  and  the  economy. 


Hie  key  to  this  plan  is  a  system  for  selecting  program  areas  that  retains  the  strong 
industry  orientation  of  the  ATP.  The  approach  is  analogous  to  that  used  in  project 
selection:  let  the  ideas  come  from  industry  and  select  on  the  basis  of  clear  evalu¬ 
ation  criteria. 
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The  criteria  for  selecting  focused  program  areas  for  ATP  investment  resemble 
those  used  for  project  selection: 

♦  potential  for  U.S.  economic  benefit; 

♦  strength  of  the  technical  ideas; 

♦  evidence  of  strong  industry  commitment;  and 

♦  the  opportunity  for  ATP  funds  to  make  a  significant  difference. 

ATP  competitions  within  the  focused  programs  will  support  specific/voyecfc, 
using  the  same  procedures  and  selection  criteria  used  for  general  competitions. 
While  most  of  the  ATP’s  resources  in  the  future  will  go  to  specific  focused  pro¬ 
grams,  the  ATP  will  continue  to  hold  general  project  competitions  open  to  any 
and  all  areas  of  technology,  holding  the  door  open  to  promising  ideas  that  don’t 
fit  into  any  current  program.  (See  Reference  2  for  a  complete  description  of  the 
process  being  used  to  define  focused  programs  and  to  solicit  program  ideas.) 


ATP  Evaluations 

From  its  rtart,  the  ATP  has  emphasized  detailed  evaluation  as  critical  to  an  effec¬ 
tive,  results-oriented  program.  The  evaluation  plan  for  the  ATP  as  a  program  has 
five  principal  elements  and  stresses  measurable  goals  vdienever  meaningful; 

4  assessing  the  ATP’s  own  critical  operafional  activities; 

♦  “portfolio”  profiles  of  applicants,  recipients,  technologies,  and 
projects; 

♦  evaluation  of  industry’s  implementation  of  both  the  R&D  and  busi¬ 
ness  components  of  ATP  projects; 

4  trading  short-term  and  intermediate  project  results;  and 

♦  measurement  of  long-term  economic  impacts. 

There  is  one  important  overriding  consideration;  The  ATP  does  not  eqiect  every 
project  or  program  to  be  a  success.  The  ATP  is  supposed  to  io&^ibigh-risk  proj¬ 
ects  that  would  not  be  undertaken  without  its  support  In  fact,  too  high  a  techni¬ 
cal  success  rate  would  suggest  that  the  project  selections  are  overly  conservative. 
Thus,  ATP’s  success  must  be  evaluated  fiom  the  perspective  of  portfolio  manr^e- 
ment,  for  \riiich  the  key  measures  are  j^gregate  returns  on  the  full  set  of  ATP- 
funded  technologies. 

Performance  of  critical  operational  activities  is  a  measure  of  the  ATP  as  a 
service  organization.  To  be  effective,  the  ATP  must  work  smoothly  with  private 


From  its  start,  the 
ATP  has  emphasized 
detailed  evaluation  as 
critical  to  an  effective, 
results-oriented  program. 
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The  ATP  sponsors 
third-party  studies  of 
award  recipients  to  get 
customer  feedback. 


industry  and,  like  industry,  must  be  agile  and  responsive  to  a  rapidly  changing  I 
technological  frontier.  Operational  activities  include: 

♦  soliciting  and  evaluating  proposals; 

♦  promoting  a  widespread  understanding  of  the  ATP  and  its 
opportunities; 

♦  providing  constmctive  feedback  to  proposers  whose  projects  were  not 
selected; 

♦  providing  additional  support  to  selected  projects,  by  initiating  links  to 
related  research  programs,  for  example;  and 

♦  monitoring  the  progress  of  projects. 

Typical  questions  include:  How  well  known  is  the  ATP  in  industry?  Is  industry 
responsive  to  ATP  solicitations?  What  steps  are  taken  to  ensure  high-quality  pro¬ 
posals  to  the  /CTP?  How  thorough  is  the  review  process?  And — very  important — 
how  do  the  program’s  immediate  customers — ^the  companies  the  ATP  works 
with — view  it? 

Considering  the  relatively  low  levek  of  funding  in  its  first  three  years  as  a  pilot 
program,  the  ATP  has  engendered  a  particulaiiy  strong  favorable  response  from 
industry.  In  four  competitions,  the  ATP  has  received  nearly  1,000  applications, 
performed  more  than  2,000  technical  evaluations  and  more  than  700  business 
evaluations  (only  the  highest-scoring  proposals  in  the  technical  evaluation  go  on  | 
to  a  business  evaluation) ,  and  made  awards  to  nearly  90  projects.  ’ 

In  addition  to  numerous  talks  and  briefings  by  ATP  personnel  around  the  coun¬ 
try,  four  conferences  have  been  sponsored  to  help  potential  applicants  with  the 
fine  points  of  the  proposal  process.  Oral  “debriefing?”  have  been  made  available 
to  all  unsuccessful  q)plicants. 

The  ATP  ^nsors  third^arty  studies  of  avard  redpients  to  get  customer  feed¬ 
back.  Comments  on  interactions  with  the  ATP  staff  have  been  uniformly  positive.  | 
For  instance,  the  vice  president  of  a  small  company  stated,  “The  personnel  within 
the  ATP  have  been  the  most  responsive  of  any  government  organization  that  I 
have  dealt  with  over  the  years.  This  is  extremely  critical.  The  commercial  markets 
in  technology-related  fields  move  very  fast,  and  a  needless  delay  can  kill  a  promis¬ 
ing  technology  or  leave  it  to  be  taken  over  by  foreign  competition.”^ 

Profiles  of  applicants,  redpients,  tedmologies,  and projects  enable  the  ATP  to 
assess  how  well  it  meets  goals  of  reaching  a  broad  spectrum  of  technologies  and 
stimulating  private  R&D.  Typical  questions  include:  Can  small  businesses  com¬ 
pete  effectively  for  ATP  awards?  How  does  the  ATP  affect  R&D  trends  in  private 
industry?  What  technologies  tend  to  receive  the  most  awards?  What  is  the  geo¬ 
graphic  distribution  of  ATP  participants?  How  do  the  ATP  projects  reflect  critical 
national  technology  goals? 
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Analyses  of  the  projects  funded  to  date  suggest  that  small  businesses  in  fact  do 
very  well  in  the  ATP.  More  than  60  percent  of  all  successful  single  applicants  are 
small  businesses,  and  small  businesses  are  playing  a  critical  role  in  18  of  the 
23  ATP  joint  ventures,  including  leading  six  of  these  projects. 

Profiles  also  suggest  that  the  ATP  has  led — as  desired — to  an  increase  in  joint 
research  and  development  ventures  in  private  industry.  In  the  first  four  competi¬ 
tions,  j^proximately  125  joint  ventures  involving  over  800  organizations  were 
formed  to  apply  to  the  ATP 

While  the  ATP  has  demonstrated  considerable  breadth  in  its  pilot  phase,  with 
projects  from  fields  such  as  manufacturing  processes,  medicine,  pollution  abate¬ 
ment,  transportation,  energy  conservation,  and  even  agricultural  pest  control, 
proposals  have  been  concentrated  in  certain  areas;  information  technologies 
(including  electronics),  advanced  materials,  manufacturing  processes,  and 
biotechnology.  ATP  projects  can  be  found  in  every  subcategory  of  the  “critical 
technologies”  list  prepared  by  the  Office  of  Science  and  Technology  Policy. 

Evaluation  of  mdustiry 's  implementation  of  both  the  R&D  and  business  com¬ 
ponents  of  ATP  projects  tracks  how  well  businesses  follow  through  on  the  busi¬ 
ness  and  commercialization  strategies  outlined  in  their  AFP  proposals.  NIST 
considers  this  an  important  element,  because  financing  research  for  its  own  sake 
is  no/  a  goal  of  the  ATP  It  also  encourages  the  award  recipient  to  constantly  re¬ 
evaluate  the  commercial  opportunities  opened  up  by  ATP  research  in  r^idly 
changing  technologies,  and  allows  ATP  managers  to  understand  how  business 
strategies  change  as  projects  evolve. 

ATP  project  manr^ers  collect  the  information  during  quarterly,  year-end,  and 
end-of-project  reviews.  The  ATP  is  field  testing  a  new,  customizable,  questionnaire 
designed  to  gather  more  detailed  data  than  are  now  available.  A  key  goal  is  to 
gather  the  information  in  a  manner  that  allows  for  easy  updating  and  minimizes 
the  r^rting  burden. 

Tracking  doort-term  and  intermediate  project  results  provides  an  indication  of 
the  ATP’s  immediate  effect  on  the  companies  that  participate.  A  number  of  meas¬ 
urable  short-term  effects  are  e:q)ected  to  provide  indicators  of  long-term  eco¬ 
nomic  success.  In  addition  to  straightforward  tracking  of  technical  milestones, 
these  indicators  include: 

♦  increased  R&D  investment  and  R&D  in  new  areas  leverj^ed  by  ATP 
funds; 

4  increased  industrial  collaborations  and  strategic  alliances; 

♦  strengthened  technological  infrastmcture  (through  the  development 
of  new  enabling  technologies)  ; 


Evaluation  of  industry’s 
implementation  of  both 
the  R&D  and  business 
components  of  ATP 
projects  tracks  how  well 
businesses  follow  through 
on  the  business  and 
commercialization 
strategies  outlined  in  their 
ATP  proposals. 
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A  third-party  survey 
of  early  ATP  award 
recipients  found  that  the 
participants  cited  the 
ability  to  pursue 
promising  lines 
of  research  that  they 
otherwise  could  not  have 
followed  as  the  most 
important  effect  of  the  ATP. 


♦  shortened  R&D  cycles; 

♦  investment  in  production  capacity;  and 

♦  productivity  improvements. 

A  variety  of  tools  are  used  for  these  analyses,  including  data  gathered  by  ATP 
project  managers,  studies  done  by  participating  companies,  and  third-party 
reviews  and  surveys  conducted  as  part  of  the  ATP  evaluation  program. 

After  four  competitions,  the  ATP  has  committed  to  $247  million,  leveraging 
approximately  $268  million  in  industry  investment  in  R&D.  Early  results  from 
this  pilot  phase  of  the  ATP  indicate  that  the  program  is  making  an  impact 

In  addition  to  the  increase  in  industrial  joint  R&D  ventures  noted  above,  a  third- 
party  survey  of  early  ATP  award  recipients  found  that  the  participants  cited  the 
ability  to  pursue  promising  lines  of  research  that  they  otherwise  could  not  have 
followed  as  the  most  important  effect  of  the  ATP.^  Forging  new  relationships  be¬ 
tween  companies,  and  between  companies  and  government  or  academic  labs, 
was  rated  by  participants  as  the  second  most  important  effect  of  the  ATP. 

Another  third-party  study  found  that  total  U.S.  R&D  work  on  advanced  technolo¬ 
gies  for  printed-wiring  boards  (PWBs)  essential  to  all  modem  electronic  devices 
more  than  quadmpled  as  a  result  of  the  ATP.^  As  a  direct  result  of  the  ATP,  m^or 
research  consortia  have  been  formed  to  pursue  advanced  technologies  in  mass 
data  storage  and  flat-^anel  di^lays,  two  technologies  considered  key  to  future  in¬ 
formation  technologies.  Other  consortia  have  been  funded  in  the  areas  of  biotech¬ 
nology,  automated  manufacturing,  and  advanced  materials. 

ATP  participants  also  have  cited  dramatic  reductions  in  development  time  and 
significant  productivity  gains — ^participants  in  the  PWB  project  estimated  aver¬ 
age  productivity  gains  of  30  percent  in  major  program  areas. 

Long-term  economic  impact  is  the  bottom  line  for  the  ATP,  and  its  measure¬ 
ment  is  key.  Program  goals  include  increased  U.S.  economic  growth,  increased  in¬ 
dustrial  competitiveness,  and  creation  of  high-value  jobs.  Measures  of  the 
long-run  success  of  the  ATP  include: 

♦  creation  of  new  industries  or  new  industrial  capabilities; 

♦  improvements  in  manufacturing  costs,  product  quality,  and  time  to 
market; 

4  increased  worldwide  maiket  share; 

4  job  creation;  and 

♦  private  and  social  rates  of  return  on  investment 

At  present,  it  is  too  early  to  measure  long-term  impacts.  Several  products  incorpo¬ 
rating  the  results  of  ATP-supported  research  have  been  introduced  or  are  near 
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commerciali2ation.  In  addition,  one  company  has  introduced  ATP  technology 
into  a  manufacturing  process  on  a  pilot  scale,  but  in  general  almost  all  ATP 
projects  ate  still  in  the  R&D  phase  of  product  development  In  most  cases,  it  will 
take  several  years  before  a  long-teim  effects  study  will  be  feasible. 

The  planned  approach  to  these  long-term  studies  is  to  use  microeconomic  case 
studies  to  estimate  specific  benefits  and  costs  of  new  technologies  developed 
under  the  ATP  This  approach  is  in  line  with  generally  accepted  economic  analy¬ 
sis  techniques,  ^lecific  projects  and  programs  for  detailed  study  will  be  selected 
using  statistical  sampling  techniques. 

The  measurement  of  long-term  economic  impacts  of  the  ATP  requires  three 
major  efforts; 

#  development  of  quantitative  measures  of  the  degree  of  influence  or 
effect  tiiat  the  ATP  has  on  the  introduction  and  diffusion  of  each  new 
technology  it  support^ 

♦  develqiment  of  quantitative  and  qualitative  measures  of  the  influ¬ 
ence  or  effect  of  each  ATP^mded  technology  on  the  econom^r,  and 

^  estimation  of  private  and  social  aggregate  economic  benefits  and 
costs  fixim  each  new  technology  developed  under  ATP  funds. 


The  planned  approach 
to  these  long-term  studies 
is  to  use  microeconomic 
case  studies  to  estimate 
specific  benefits  and  costs 
of  new  technologies 
developed  under  the  ATP. 
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NIST  LABORATORY 
PROGRAMS 


As  noted  in  a  recent 
report  by  the  Office  of 
Technology  Assessment, 
NIST's  laboratory  program 
occupies  a  ^‘unique 
niche  in  the  nation’s 
infrastructure.” 


NIST’s  eight  laboratories  serve  all  sectors  of  U.S.  industry  through  tightly 
focused  research  programs  and  services  that  address  industry’s  needs  for 
measurement  and  infrastructural  technology.  Industry  traditionally 
underinvests  in  the  development  of  these  infrastructural  technolo^es  because 
they  are  used  simultaneously  by  many  firms  and  typically  are  not  embodied  in 
products,  making  it  difficult  for  individual  firms  and  even  industries  to  recover 


R&D  investments.  However,  measurement  methods,  evaluated  data,  process  mod¬ 
els,  interface  standards,  and  other  types  of  infrastructural  technologies  are  pacing 
factors  in  technology  development  and  application,  setting  the  upper  limit  on 
what  can  be  accomplished  in  the  laboratory  or  on  the  factory  floor. 


As  noted  in  a  recent  report  by  the  Office  of  Technology  Assessment,  NISTs  labora¬ 
tory  program  occupies  a  “unique  niche  in  the  nation’s  infrastructure.”  The 
report  also  characterized  the  program’s  cote  competency  and  preferred  problem¬ 
solving  approach: 

NIST  has  earned  a  worldwide  reputation  for  impartiality  and  techni¬ 
cal  excellence.  Its  competencies  in  metrology — ^the  science  of  meas¬ 
urement — span  a  number  of  disciplines.  The  efficiency  of  solving  a 
measurement  problem  once  at  NIST  and  then  disseminating  the  re¬ 
sults  throughout  the  vdiole  industry,  rather  than  each  company  per¬ 
forming  the  job  independently  for  itself,  provides  outstanding 
leverage  for  NISTs  metrological  development^ 

NISTs  evaluations  of  industry’s  technology  needs  indicate  widespread  demand 
for  enhanced  measurement  c^abilities,  and  industry’s  own  analyses  concur.  For 
example,  a  1993  assessment  by  the  Semiconductor  Industry  Association  (SIA) 
identified  unmet  measurement  needs  as  impeding  the  U.S.  semiconductor  indus¬ 
try’s  progress  toward  accomplishing  critical  technology  goals.  SIA  called  on  NIST 
for  increased  assistance,  describing  the  agency  as  the  “only  place  in  the  U.S. 
where  the  broad  range  of  measurements  needed  for  semiconductor  processing  are 
routinely  and  systematically  developed.” 


Similarly,  the  Council  on  Radiation  Measurements,  which  represents  more 
than  150  U.S.  companies,  asked  NIST  to  improve  the  accuracy  of  measurement 
standards  for  optical  and  infrared  radiation  by  a  factor  of  10.  The  request  was 
motivated  by  the  companies’  desire  to  improve  quality  control  in  a  variety  of 
manufacturing  processes.  The  collaboration  has  produced  a  cryogenic 
radiometer  that  is  possibly  the  most  accurate  in  the  world  and  is  being  tested  in  a 
number  of  sqiplications. 
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Setting  Laboratory  Priorities 

NIST’s  laboratories  set  their  priorities  in  consultation  with  industry  in  accordance 
wth  six  guiding  criteria: 

♦  the  magnitude  and  immediacy  of  industrial  need; 

♦  the  degree  of  correspondence  between  a  particular  industrial  need 
and  NISTs  mission  to  develop  infrastmctural  technologies; 

♦  the  opportunity  for  NIST  participation  to  make  a  major  difference; 

♦  the  nature  and  size  of  the  anticipated  impact  resulting  from  NISTs 
participation; 

♦  NISTs  capability  to  respond  in  a  timely  fashion  with  a  high-quality 
solution;  and 

♦  the  nature  of  opportunities  afforded  by  recent  advances  in  science  and 
technology. 

NISTs  laboratories  try  to  anticipate  the  measurement  and  other  infrastmctural 
technology  needs  of  industry.  The  ideal  is  to  have  solutions  available  before  pro¬ 
spective  problems  and  challenges  materialize  as  actual  obstacles  in  product  devel¬ 
opment,  manufacturing,  maritet  transactions,  or  other  industrial  and  business 
activities.  If  NIST  does  not  respond  early  to  looming  technolo^cal  hurdles,  the 
effectiveness  of  its  laboratories  is  diminished.  Therefore,  failures  to  anticipate 
major  technology  needs  also  must  be  taken  into  account  as  the  laboratories’ 
priority-setting  performance  is  reviewed. 

In  their  strategic  planning,  the  laboratories  employ  a  variety  of  formal  and  infor¬ 
mal  mechanisms  for  soliciting  industry  input  and  gauging  its  priorities.  Formal 
mechanisms  include  NIST-convened  conferences  devoted  to  eliciting  and  synthe¬ 
sizing  company  and  industry-wide  views  on  key  technical  challenges  and  major 
goals  in  important  technology  areas.  In  1993, 40  priority-setting  conferences 
were  convened  to  address  topics  ranging  fiom  “green”  manufacturing  technolo¬ 
gies  to  technical  issues  in  network  security. 

Over  the  past  two  years,  a  series  of  conferences  focused  on  measurement  needs 
arising  from  tightening  dimensional  tolerances  in  manufactured  products  from 
integrated  circuits  to  aircraft.  Responding  to  the  needs  voiced  by  representatives 
of  the  $15  billion  U.S.  gear  manufacturing  industry,  NIST  joined  with  the  Ameri¬ 
can  Society  of  Mechanical  Engineers  and  the  Department  of  Energy’s  Oak  Ridge 
Y-12  plant  to  create  a  NIST/DOE  Center  for  Gear  Metrology  at  the  Y-12  facility. 

The  unprecedented  interagency  collaboration  is  now  underway. 

Other  formal  mechanisms  for  assessing  industry  priorities  include  the  agency’s 
Visiting  Committee  on  Advanced  Technology,  an  advisory  body  primarily  com¬ 
posed  of  industry  representatives.  Industry  also  is  well  represented  on  independent 


In  their  strategic  planning, 
the  laboratories  employ  a 
variety  of  formal  and 
informal  mechanisms  for 
soliciting  industry  input 
and  gauging  its  priorities. 
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Each  laboratory 
has  cultivated  strong 
working  relationships 
with  industrial,  trade, 
and  professional 
organizations  in  its 
areas  of  technology 
concentration. 


assessment  panels.  Origanized  by  the  National  Research  Council,  these  panels 
annually  review  the  performance  of  each  NIST  laboratory  and  evaluate  its  short- 
and  long-term  goals. 

Each  laboratory  has  cultivated  strong  working  relationships  with  industrial, 
trade,  and  professional  organizations  in  its  areas  of  technology  concentration. 
The  program  of  NIST’s  Building  and  Fire  Research  Laboratory,  for 
example,  is  guided  by  a  prioritized  research  agenda  developed  by  volunteer 
e}q)erts  from  the  building  and  fire  communities  under  the  auspices  of  the 
National  Institute  of  Building  Sciences.  NIST  personnel  also  participate  actively 
in  industry-organized  technology-planning  exercises.  Recent  examples  include 
the  Semiconductor  Industry  Association’s  initiative  to  develop  a  comprehensive 
technology  road  map  and  the  21st  Century  Manufacturing  Enterprise  Strategy 
Project^ 

The  laboratories  also  solicit  industry  input  through  formal  surveys  and  through 
reviews  of  NIST-prepared  planning  documents  and  needs  assessments.  In  early 
1993,  the  Electronics  and  Electrical  Engineering  Laboratory  issued  a  definitive 
assessment  of  measurement  needs  in  nine  fields  of  electronics  technology.  Devel¬ 
oped  in  consultation  with  industry,  the  448-page  document,  Measuremenisfor 
Competitiveness,  identifies  high-impact  measurement  capabilities  that  are 
widely  needed  by  the  U.S.  electronics  industry  but  are  beyond  the  resources  of 
individual  companies  to  develop.*  It  senes  as  ejqrlicit  guidance  for  setting  the 
laboratory’s  priorities. 

Other  mpans  of  assessing  industry’s  infiastructural  technology  needs  include; 

^  visits  to  companies; 

♦  participation  on  mote  than  800  national  and  international  standards 
committees; 

♦  measurement  “round  robins,”  which  provide  a  comparative  basis  for 
assessing  the  current  state  of  measurement  practices  employed  by 
industry  and  for  identifying  key  technical  problems; 

♦  joint  demonstrations  and  experiments  to  identify  problems  and 
requirements  for  supporting  technology; 

♦  regular  researcher-to-researcher  interactions;  and 

♦  participation  in  consortia,  including  the  13  cooperative  arrangements 
organized  by  NIST  and  those  organized  by  other  groups,  such  as 
PDES  Inc.  and  the  Semiconductor  Research  Corp. 


NIST  has  found  it  necessary  to  use  this  complete  set  of  tools  to  determine 
industrial  needs.  Woritshops,  conferences,  and  surveys  provide  valuable 
information  on  an  industry,  but  this  information  usually  is  not  complete  or 
cific  enough  to  be  the  sole  basis  for  program  planning.  In  turn,  company  visits, 
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participation  on  standards  committees,  “round  robins,”  researcher-to-researcher 
interactions,  and  participation  in  consortia  offer  insight  into  ^ific  company 
needs,  and  they  afford  the  additional  advantage  of  directly  involving  NIST  re- 
searcheis,  vdio  provide  their  peers  from  industry  with  a  direct  link  to  the  agency’s 
programs  and  projects.  Thus,  combining  information  that  is  ^ific  to  firms  and 
information  that  can  be  generalized  to  an  industry  or  even  groups  of  industries  is 
necessary  to  develop  a  balanced  view  of  challenges  facing  the  private  sector  and 
of  the  relative  importance  of  each  challenge.  To  assure  that  the  proper  balance 
has  been  reached,  NIST  publishes  its  planning  documents  for  review  by  outside 
panels  and  by  industry. 

This  planning  process  is  continuous,  involves  most  of  NISTs  professional  staff, 
and  provides  a  wealth  of  useful  information.  In  developing  a  new  program  in 
magnetic  engineering,  for  example,  staff  of  the  Chemical  Science  and  Technol¬ 
ogy  Laboratory  solicited  the  views  of  e^rts  at  mote  than  50  companies  and 
universities  to  identify  the  critical  technical  obstacles  perceived  as  impeding  the 
development  and  application  of  new  thin-film  magnetic  materiab. 

Another  example  demonstrates  how  this  approach  is  used  in  planning  consortia. 
In  1991,  members  of  the  Materials  Science  and  Engineering  Laboratory  visited 
aerospace  industry  companies  to  acquire  first-hand  information  on  problems  en¬ 
countered  in  the  precision  casting  of  metal  alloys.  Analysis  of  this  information 
guided  planning  for  a  meeting,  co-^nsored  by  the  Aerospace  Industries  Associa¬ 
tion,  to  explore  the  merits  of  forming  a  consortium  to  improve  the  precision 
casting  process.  A  subsequent  meeting,  attended  by  industry,  university,  and  gov¬ 
ernment  representatives,  resulted  in  a  detailed  technical  research  plan  addressing 
issues  in  key  processing  areas  and  defining  deliverables  in  forms  usable  by  indus¬ 
try.  Subsequently  refined,  that  plan  is  now  being  carried  out  by  a  NIST-led  consor¬ 
tium  involving  seven  manufacturers,  seven  universities,  and  three  federal 
laboratories.  The  effort  is  a  distributed,  cooperative  undertaking,  with  NIST  and 
other  members  perfonning  in-house  research  to  accomplish  agreed-upon  tasks 
that  will  contribute  to  accomplishing  the  consortium’s  four  major  objectives. 

NISTs  iterative  approach  to  planning  and  priority  setting  enables  it  to  re^nd 
quickly  to  new  developments  and  opportunities.  A  clear  picture  of  industry’s  needs 
also  helps  laboratory  managers  to  determine  vdien  it  is  appropriate  to  terminate 
programs  and  reallocate  scarce  resources  to  address  new,  higher  priority  problems 
that  have  emerged.  For  instance,  the  Physics  Laboratory  terminated  a  project  to 
provide  reaction-rate  data  for  fusion  reactors  and  reassigned  the  technical  staff  to 
work  on  plasma-processing  issues  relevant  to  semiconductor  manufacturing, 
which  led  to  a  productive  collaboration  with  SEMATECH. 


NISrs  iterative 
approach  to  pianning 
and  priority  setting 
enabies  it  to  respond 
quickly  to  new 
developments  and 
opportunities. 
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Short-  to  Medium-Term 
Measures  of  Performance 


Customer  feedback, 
gathered  through  a 
variety  of  mechanisms, 
is  the  laboratories’ 
principal  source  of 
evaluative  information. 


NIST  laboratories  use  a  variety  of  measures  to  track  and  evaluate  performance, 
including  the  value  and  utility  of  research  deliverables  and  services.  Customer 
feedback,  gathered  through  a  variety  of  mechanisms,  is  the  laboratories’  princi¬ 
pal  source  of  evaluative  infoimation. 

Although  each  laboratoiy  has  its  own  procedures  for  monitoring  performance 
and  technical  progress — and  not  all  of  the  laboratories  use  every  available 
tool — all  set  goals  for  individual  projects,  determined  on  the  basis  of  perceived 
customer  needs  identified  during  planning  and  priority  setting.  Technical  mile¬ 
stones  are  established  for  individual  projects,  and  progress  is  evaluated  internally 
on  at  least  a  quarteriy  basis.  Customer  feedback  also  is  analyzed  during  project, 
group,  division,  and  laboratory  reviews,  not  only  to  assess  performance  and  rates 
of  technical  progress  but  also  to  identify  changes  in  customer  needs  that  may  war¬ 
rant  redirecting  laboratory  resources.  Besides  their  own  internal  reviews  and  those 
conducted  by  NIST  management,  all  laboratories  undergo  annual  assessments  by 
external  panels  convened  by  the  National  Research  Council.  These  assessment 
panels  produce  written  evaluations  of  performance,  missions,  and  short-  and 
long-term  goals — ^for  the  laboratory  overall  and  for  each  division. 

NIST  gathers  information  on  its  primary  “products” — ^measurements,  standards, 
dataha.SK,  process  models,  and  the  other  types  of  deliverables  produced  by  its  labo¬ 
ratory  program.  These  deliverables  usually  take  the  form  of  technical  informa¬ 
tion  that  NIST  makes  widely  available  to  U.S.  industry.  Measures  of  the  relevance 
and  value  of  this  information  to  industry  include: 

♦  industry  attendance,  comments,  and  level  of  participation  at  techni¬ 
cal  woritshops; 

♦  number  of  inquiries  and  requests  for  information; 

♦  attendance  at  technical  training  sessions  provided  by  NIST  persormel; 

♦  commercialization  of  products  incorporating  the  results  of  NIST 
R&D;  and 

♦  2q)plication  of  NIST  R&D  results  to  industrial  processes. 

For  federal  laboratories,  a  widely — and,  perhaps,  overly — reported  process 
metric  is  the  number  of  cooperative  R&D  agreements  (CRADAs)  that  laboratories 
have  entered  into  with  U.S.  businesses.  Because  NIST  has  been  working  with  in¬ 
dustry  for  more  than  90  years,  the  agency  was  quick  to  embrace  CRADAs  as  an  ad¬ 
ditional  tool  for  working  with  industry.  Across  the  federal  government,  NIST  has 
the  highest  ratio  of  CRADAs  per  number  of  technical  staff  and,  by  far,  the  shortest 
average  time  for  processing  agreements.  According  to  Scierux  magazine,  “Only 


B-2-18 


the  National  Institute  of  Standards  and  Technology,  created  to  woric  with  the  pri¬ 
vate  sector,  appears  to  be  doing  what  Congress  intended.  ...[0]nlv  NIST  appears  to 
have  manned  to  embrace  CRADAs  without  getting  smothered.”^ 

CRADA  tallies,  however,  are  an  incomplete  measure  of  performance — ^just  as  the 
numbers  of  patents  received  and  licenses  issued  do  not  capture  a  technology  or¬ 
ganization’s  full  technical  output  and  level  of  innovation.  CRADAs  represent  but 
one  mechanism  for  working  with  and  addressing  the  technology  needs  of  U.S.  in¬ 
dustry — effective  for  accomplishing  some  technical  objectives,  but  injq)propriate 
for  others.  In  fact,  most  forms  of  NIST-provided  technical  assistance  are  non¬ 
proprietary  (used  by  many  firms)  and  are  best  accomplished  without  the  formal¬ 
ity  of  a  legal  contract 

Other  measures  and  information  used  to  assess  industrial  relevance,  perform¬ 
ance,  and  productivity  of  laboratory  activities  include: 

♦  level  of  industry  commitment  to  NIST  projects  and  consortia 
(e.g.,  number  of  participating  companies,  number  of  visiting 
researchers  assigned  to  NIST,  value  of  resources  committed); 

♦  number  of  guest  researchers  ftora  industry; 

4  extent  of  NISTs  contributions  to  industry’s  voluntary  standards  (e.g., 
number  of  standards  incorporating  NISTs  work,  number  of  member¬ 
ships  on  standards  committees) ; 

♦  number  of  joint  industry-NIST  “round-robins”;  and 

♦  number  of  repeat  customers. 


Measures  of  Long-Term  Impact 

In  the  early  1980s,  NIST  initiated  a  series  of  periodic  assessments  of  the 
economic  impacts  of  NIST  research.^®"^^  Conducted  by  independent 
researchers  under  contract  to  NIST,  these  third-party  assessments  have  estimated 
the  aggregate  rates  of  return  (also  referred  to  as  sodal  rates  of  return)  on  work 
addressing  infrastructural  technology  needs  of  industry.  Returns  on  NIST  woric  in 
the  six  technology  areas  evaluated  thus  far  range  fiom  63  percent  to  423  percent, 
greatly  exceeding  the  average  rate  of  return  on  private-sector  innovations  and  the 
rates  reported  in  the  few  studies  of  other  government  research  programs.  To 
economists,  this  disparity  implies  underinvestment  in  R&D  aimed  at  developing 
measurement  methods  and  other  infrastructural  technologies. 

An  example  is  NISTs  work  supporting  the  U.S.  qrtical  fiber  industry.^^  NIST- 
developed  measurement  technologies  served  as  the  basis  for  more  than  20  indus¬ 
try  standards  that  have  helped  reduce  market  transaction  costs  arising  from 
disagreements  between  optical  fiber  manufacturers  and  their  customers.  The 
standards  established  a  solid  basis  for  evaluating  the  technical  performance  of 


Third-party  assessments 
have  estimated  the 
aggregate  rates  of  return 
on  work  addressing 
infrastructural  technology 
needs  of  industry. 
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NIST  also  is  placing 
increased  emphasis 
on  tracking  how 
companies  and  industries 
use  specific  services 
and  the  results  of  specific 
R&D  projects. 


fibers.  The  social  rate  of  return  was  estimated  to  exceed  400  percent,  and  one 
manufacturer  has  credited  the  standards  with  significantly  expanding  the  size  of 
the  market  for  optical  fibers. 

In  addition  to  the  series  of  case  studies  of  economic  impact,  which  is  continuing, 
NIST  also  is  placing  increased  emphasis  on  tracking  how  companies  and  indus¬ 
tries  use  specific  services  and  the  results  of  ^lecific  R&D  projects.  For  example, 
the  Siting  Committee  on  Advanced  Technology  studied  the  commercialization 
of  NIST  innovations  that  have  won  “R&D  100  Awards"  in  the  annual  competi¬ 
tion  sponsored  by  Research  and  Devebpment  magazine.  Between  1973  and 
1990,  NIST  won  71  R&D  100  Awards;  about  three-fifths  of  the  innovations  (41) 
achieved  commercial  impact  NIST  also  has  begun  to  prepare  a  series  of  brief 
“industrial  impact  statements”  that  are  intended  to  c^ture  the  nature  of  the 
laboratories’  assistance  to  companies  or  industrial  sectors,  including  resulting 
improvements  in  products  or  services,  processes,  and  market  performance,  as  well 
as  jobs  created. 
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MANUFACTURING 
EXTENSION  PARTNERSHIP 


The  Manufacturing  Extension  Partnership,  or  MEP,  is  a  growing  nation¬ 
wide  networic  of  manufacturing  extension  services  that  provides  small  and 
medium-sized  U.S.  manufacturers  with  technical  assistance  as  these  firms 
modernize  their  operations  to  increase  their  competitiveness.  Comprising  the 
core  of  the  nation’s  manufacturing  base,  the  more  than  370,000  U.S.  manufac¬ 
turing  establishments  that  employ  fewer  than  500  people  often  are,  as  charac¬ 
terized  in  a  recent  National  Research  Council  study,  “operating  below  their 
potential.  Their  use  of  modem  manufacturing  equipment,  methodologies,  and 
management  practices  is  inadequate  to  ensure  that  American  manufacturing 
will  be  globally  competitive.”^^ 

A  variety  of  obstacles  hinder  small  and  medium-sized  manufacturers’  modern¬ 
ization  efforts.  Those  that  impede  the  adoption  of  appropriate  modem  technology 
and  organizational  methods  include  lack  of  resources  and  in-house  technical 
expertise,  limited  awareness  of  changing  manufacturing  technology  and  its  appli¬ 
cations,  and  difficulty  in  locating  unbiased  sources  of  information  and  technical 
assistance.  Viewed  fiom  a  national  perq)ective,  currently  available  public  and 
private  sources  of  assistance — which  range  fiom  large  coii^anies’  supplier- 
improvement  programs  to  small,  local  industrial  outreach  programs  and  private- 
sector  consultants — are  fragmented  and  vary  greatly  in  breadth  and  depth  of 
services.  Collectively,  they  reach  only  a  small  fiaction  of  small  and  medium-azed 
manufacturers  whose  existence  is  threatened  by  continued  reliance  on  outdated 
technology,  production  techniques,  and  management  practices. 

By  providing  leadership  and  building  a  national  framework  for  the  ddivery  of 
manufacturing  extension  services,  NISTs  MEP  will  organize  a  comprehensive, 
yet  locally  re^nsive,  system  to  help  small  and  medium-sized  manufacturers 
upgrade  their  equipment,  techniques,  and  operations.  From  a  base  of  seven 
regional  Manufacturing  Technology  Centers  (MTCs),  established  between  1989 
and  1992,  the  MEP  is  planned  to  grow  to  over  100  extension  centers  by  1997. 
Linked  by  a  coordinating  national  infiastmcture,  each  center  and  each  partner¬ 
ing  organization  will  be  an  entry  point  into  an  integrated  network  of  technical 
resources,  services,  and  expertise  on  topics  ranging  fiom  computer-aided  design 
and  manufacturing  to  just-in-time  inventory  methods  to  workforce  training. 

The  evolving  MEP  network  has  several  components: 

♦  regional  MTCs,  or  MTC-like  service  providers,  located  in  areas  of  high 
manufacturing  density. 


By  providing  leadership 
and  building  a  national 
framework  for  the 
delivery  of  manufacturing 
extension  services, 

NIST’s  MEP  will  organize 
a  comprehensive,  yet 
locally  responsive, 
system  to  help  small 
and  medium-sized 
manufacturers. 
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♦  Manufacturing  Outreach  Centers,  or  MOCs,  which  serve  areas  of  lower 
manufacturing  density,  either  as  free-standing  entities  or  as  MTC 
satellites; 

♦  the  State  Technology  Extension  Program,  or  STEP,  which  provides 
funding  and  technical  support  for  planning,  implementation,  and 
regional  linkages  to  strengthen  industrial  extension  efforts  in  the 
states;  and 

♦  “LINKS,”  which  encompasses  the  national  stmcture  of  communica¬ 
tions,  data  systems,  evaluation,  field-agent  training,  tool  develop¬ 
ment,  and  linkages  with  technology  sources. 

NIST’s  MEP  staff  have  played  a  key  role  in  the  review  and  merit-based  selection 
of  winners  of  deployment  awards  made  under  President  Clinton’s  Technology 
Reinvestment  Project  (TRP),  vdiich  is  managed  by  the  Defense  Department’s 
Advanced  Research  Projects  Agency.  NIST  is  managing  42  of  the  70  TRP  deploy¬ 
ment  activities  selected  to  date,  and  these  activities  will  be  incorporated  into  the 
manufacturing  extension  networic.  Recent  TRP  award  winners  include  22  new 
centers  that  will  enable  NIST  to  accelerate  development  of  the  MEP  networic. 


Recipients  of  MEP 
funding  are  selected 
through  merit-based 
competitions  that  assess 
how  fully  each  candidate 
satisfies  the  criteria 
established  for  each 
component  of  the  network. 


Setting  Priorities 

Guiding  Principles.  The  design  and  evolution  of  the  national  manufacturing 
extension  network  are  guided  by  several  basic  principles  that  broadly  define  the 
MEP’s  priorities,  (^rational  and  o^anizational  philosophies,  scq*  of  services 
and  activities,  and  strategic  emphases. 

First,  all  recipients  of  MEP  funding  are  selected  rmrit-based  competi¬ 

tions  that  assess  how  fully  each  candidate  satisfies  the  criteria  established  for 
each  component  of  the  network. 

Second,  service  providers  in  the  MEP  network  tailor  assistance  and  technology 
elutions  to  the  needs  and  constraints  of  client firms,  including  budgets  and 
worirforce  cr^abilities.  The  providers  are  client-driven,  not  providernlriven. 

Third,  extension  centers  and  other  components  of  the  MEP  network  are  not 
supported  to  perform  manufacturing  R&D,  rancentrating  entirely  on  the 
diffusion  of  ^propriate  technologies  to  speed  industrial  modernization. 

Fourth,  the  MEP  is  mdusive,  and  it  prevents  duplication  of  effort  by  emphasiz¬ 
ing  a  networic  design  that  takes  maximum  advant^e  of  the  knowledge,  experi¬ 
ence,  and  expertise  of  a  broad  spectrum  of  existing  organizations. 

This  principle  yields  two  other,  closely  related  characteristics  that  are  key: 

♦  The  network  is  non-hierarchical.  It  builds  on  grassroots  proposals  mo¬ 
tivated  by  and  tailored  to  the  needs  of  local  industry  and  allows  local 
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flexibility  and  innovation  in  addressing  those  needs,  while  providing 
the  technical  support  and  resources  of  a  national  infrastructure. 

♦  The  MEP  does  not  compete  with  private  technical  consultants;  it 
recognizes  the  important  role  that  these  consultants  play  in  helping 
small  and  medium-sized  manufacturers  improve  their  operations. 

The  NIST  MTCs  have  developed  constructive  relationships  with  pri¬ 
vate  consultants,  who  can  help  the  centers  extend  their  reach  and 
broaden  the  range  of  technical  assistance  and  expertise  accessible 
through  the  MTCs.  At  the  same  time,  in  many  cases,  MTC  activities 
help  private  consultants  expand  their  reach  by  qualifying  small 
manufacturers  and  helping  them  understand  the  value  of  consulting 
services.  To  ensure  a  clear  understanding  of  this  relationship  and  to 
avoid  potential  conflicts,  MEP  staff  are  now  evaluating  how  private 
consultants  serve  small  manufacturers  and  what  role  the  MTCs  play 
in  that  process. 

Selection  Criteria.  Separate  selection  criteria  have  been  developed  for  each 
MEP  component,  although  there  is  substantial  overlap  because  of  shared  objec¬ 
tives.  All  centers  and  projects  are  selected  only  after  undergoing  competitive  re¬ 
view.  The  criteria  used  to  evaluate  proposals  for  new  centers  are  representative: 

♦  Knowledge  of  target firm:  Comprehensive  understanding  of  an 
area’s  manufacturing  base,  including  business  size,  industry  types, 
product  mix,  and  technology  requirements. 

4  Linkages  to  external  sources  of  technology,  in¬ 

cluding  educational  institutions,  state  and  regional  technology  trans¬ 
fer  programs,  and  federal  laboratories. 

♦  Ddivery  medoanism:  Tlie  staff,  resources,  and  methodology  for 
effectively  delivering  appropriate  manufacturing  technologies  and 
techniques  to  local  or  regional  manufacturers,  as  well  as  the  ability  to 
form  effective  partnerships  with  other  organizations  offering  comple¬ 
menting  services  or  resources. 

♦  Management  and financial  plan:  An  effective  management  struc¬ 
ture,  full-time  top  management,  additional  sources  of  funds,  and  an 
effective  system  of  internal  evaluation. 

Program  Evaluation 

In  preparing  for  the  MEP’s  expansion,  staff  members  are  developing  a  strategic 
plan  that  integrates  the  functions  of  each  service-delivery  component  of  the  net¬ 
work  and  lays  the  methodological  foundation  for  evaluating  program  perform¬ 
ance  and  measuring  results.  A  national,  standardized  system  of  data  collection 


MEP  builds  on  grassroots 
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a  national  infrastructure. 
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A  national,  standardized 
system  of  data  collection 
and  evaluation 
will  enable  MEP  staff  to 
assess  the  efficiency  and 
effectiveness  of 
the  entire  network. 


and  evaluation  will  enable  MEP  staff  to  assess  the  efficiency  and  effectiveness  of 
the  entire  network,  each  network  component,  each  type  of  extension  service,  and 
each  service  provider.  The  evaluation  system  will  identify  the  services  most  valued 
by  small  and  medium-si2ed  manufacturers;  characterize  the  types  of  finiis  and 
industries  that  make  the  greatest  use  of  manufacturing  extension  services,  as  well 
as  those  that  make  the  least  use  of  services;  prevent  duplication  of  effort;  and 
measure  economic  impact  at  levels  ranging  from  individual  services  and  centers 
to  the  MEP  as  a  whole. 

Measures  of  Organizational  Performance.  All  MEP  organizations  are  re¬ 
quired  to  submit  detailed  quarterly  and  annual  reports,  standardized  for  each  of 
the  four  components  of  the  networit,  and  they  undergo  annual  performance  re¬ 
views  by  MEP  regional  managers.  At  the  end  of  the  third  year  of  operation,  exten¬ 
sion  centers  undergo  rigorous  evaluations  by  outside  panels  appointed  by  NIST. 

Reported  quarterly  by  each  center,  the  following  types  of  information  help  MEP 
management  to  evaluate  overall  performance  of  the  MEP  organizations  and  to 
assess  market  penetration  (i.e.,  number  of  client  firms  relative  to  the  size  of  the 
potential  client  base),  the  mix  of  services  provided,  the  breadth  and  depth  of 
organizational  linkages,  and  the  ability  of  the  center  to  generate  revenue  to 
support  operations: 

♦  staff  composition  and  percentage  of  employees  working  with 
clients  in  the  field; 

♦  number  of  establishments  served,  by  size; 

♦  ratio  of  the  number  of  technical  assistance  projects  proposed  to  client 
companies  to  the  number  initiated; 

♦  number  of  activities  initiated,  by  type  of  activity  (e.g.,  formal 
assessment,  technical  assistance  project,  initial  site  visit,  information 
referral),  technical  focus  (e.g.,  control  systems;  plant  layout; 
computer-aided  design,  manufacturing,  or  en^eering),  and  size  of 
firm; 

♦  number  of  participants  and  number  and  types  of  firms  represented  at 
training  programs  and  other  events;  and 

♦  results  of  client  valuation  surveys. 

Upon  completing  technical  assistance  projects,  the  centers  ask  client  firms  to 
assign  values  to  the  services  provided.  Clients  estimate  the  impact  that  the  just- 
concluded  project  will  have  on  company  performance  over  the  next  12  months. 

Specifically,  firms  are  asked  to  estimate  the  anticipated  impact  of  the  project  on 
sales,  costs,  cjqrital  spending,  inventory  levels,  and  jobs  (number  created  or 
saved).  Although  subjective,  these  measures  provide  center  management  and 
personnel  with  the  means  to  ascertain  levels  of  customer  satisfaction  and  to 
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differentiate  among  clients’  perceptions  of  the  type  and  magnitude  of  benefits 
attributable  to  center  activities. 


MEP  management  fully  ^preciates  the  importance  of  measuring  performance, 
but  it  also  recognizes  the  need  for  flexibility  in  reporting  formats,  particularly  for 
pilot  projects  that  are  testing  experimental  services  and  service-delivery  mecha¬ 
nisms.  Overly  rigid  process  and  outcome  measures  could  yield  data  that  provide 
little  meaningful  infonnation  for  evaluations  of  perfonnance  and  impact  and,  at 
the  same  time,  inhibit  experimentation  and  innovation.  Successful  “LINKS”  and 
extension  enabling  TRP  projects,  for  example,  may  provide  services  that 
indirectly  affect  manufacturers  but  directly  improve  the  quality  and  accessibility 
of  information  and  services  provided  by  extension  centers.  Measures  of  perform¬ 
ance  and  results  are  being  developed  for  these  kinds  of  activities,  with  the  recogni¬ 
tion  that  what  gets  measured  gets  emphasized  and  that  not  all  measures  are 
qually  significant  for  all  types  of  activities  and  MEP  con^nents. 

Measures  of  Economic  Impact  In  addition  to  evaluating  the  performance 
of  each  extension  center,  NIST  also  measures  each  center’s  specific  intact  on  cli¬ 
ent  companies.  In  contrast  to  the  ATP  and  NISTs  laboratory  programs,  the  MEP 
is  e^qiected  to  generate  economic  benefits  within  a  short  time  span.  Indeed,  MTC 
client  firms  reported  substantial  benefits,  including  cost  savings,  higher  produc¬ 
tivity,  and  increased  earnings,  within  the  first  year  of  each  center’s  operation. 

Though  necessarily  small  in  scope  because  of  the  limited  scale  of  manufacturing 
extension  efforts  undertaken  thus  far,  the  results  of  analyses — supplemented  by 
case  studies  and  other  anecdotal  evidence — ^Ulustrate  the  large  potential  for  sig¬ 
nificant  economic  and  company  benefits.^^^®  Cumulatively  valued  and  based 
on  self-reported  data  horn  MTC  client  firms,  company-realized  benefits  from  for¬ 
mal  MTC  technical  assistance  projects  totaled  an  estimated  $320  million  between 
1989  and  1992,  translating  into  a  return  of  over  $7  on  each  federal  dollar  in¬ 
vested  in  the  centers.  This  estimate  undervalues  the  impact  of  some  MTC  services, 
such  as  seminars  and  information  referrals.  Many  firms  consider  these  “soft” 
services  to  be  very  beneficial,  even  though  their  impact  is  difficult  to  quantify. 

For  the  network’s  individual  extension  centers,  as  well  as  for  the  MEP  as  a  vliole, 
the  fundamental  unit  of  analysis  is  and  will  continue  to  be  individual  firms.  On 
an  annual  basis,  the  MTCs  also  administer  questionnaires  to  ascertain  the  prog¬ 
ress  made  by  client  firms.  One  year  after  the  completion  of  a  technical  assistance 
project  or  other  substantive  interaction  with  a  center,  client  firms  are  asked  to  as¬ 
sess  their  progress  and  business  health  by  comparing  information  in  key  perform¬ 
ance  areas  one  year  before  and  one  year  after  projects  have  been  completed. 

The  before-and-after  assessment  considers  three  levels  of  outcomes,  vdiich  will  be 
analyzed  in  relation  to  a  “control”  group. 


MTC  client  firms  reported 
substantial  benefits, 
including  cost  savings, 
higher  productivity,  and 
increased  earnings,  within 
the  first  year  of  each 
center’s  operation. 
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Base-level  client 
outcomes  serve  as 
indicators  of  how 
successfully  centers  are 
assisting  firms  in 
adopting  appropriate 
technologies  and 
management  practices. 


Base-level  client  outcomes  serve  as  indicators  of  how  successfully  centers  are 
assisting  firms  in  adopting  appropriate  technologies  and  management  practices; 
making  more  use  of  capital,  labor,  and  materials;  and  improving  response  time 
and  other  key  aspects  of  manufacturing  performance.  These  measures  are: 

♦  changes  in  scr^  rate  (value  of  scrap  per  sales — an  indicator  of 
quality  and  of  efficiency  of  material  use); 

♦  changes  in  computer  use  (percent  of  employees  using  computers  or 
programmable  machine  controllers  at  least  weekly — an  indicator  of 
broadening  applications  of  computers,  the  essential  element  of  many 
new  manufacturing  and  business  technologies);  and 

♦  changes  in  inventory  turns  (ratio  of  sales  to  inventory — an  indicator 
of  increasing  throughput,  which  may  be  the  result  of  improved  lay¬ 
out,  scheduling,  and  routing). 

Intermediate-level  client  outcomes  are  designed  to  reflect  changes  that  stem 
horn  performance  improvements  at  the  base  level  of  operations,  although  these 
changes  may  take  longer  to  materialize.  These  include: 

♦  changes  in  the  ratio  of  sales  per  employee — an  indicator  of 
increased  labor  productivity;  and 

♦  f:hangps  in  manufacturing  lead  time — an  indicator  of  decreased 
re^nse  time  to  customer  orders. 

Tcp-level  client  outcomes  reflect  business  and  employment  gains  traceable,  at 
in  part,  to  operational  improvements  cjqrtured  by  the  previous  categories  of 
measures.  Among  the  outcomes  measured  are: 


4  change  in  total  sales, 

♦  change  in  eiq)ort  sales, 

♦  change  in  enqiloyment,  and 

♦  change  in  employee  income  payroll  per  employee) . 

Although  these  objective  outcome  measures  may  not  correspond  directly  to  the 
extension  services  provided,  they  are  easily  tracked  by  client  firms,  and  they  have 
the  virtue  of  being  related,  at  least  indirectly,  to  the  substance  and  aims  of  most 
service  activities.  As  they  come  on  line,  all  MEP  service  providers  will  be  required 
to  submit  client  progress  reports  based  on  surveys  to  ascertain  the  performance  of 
firms  in  the  years  before  and  after  completion  of  an  extension  service  activity. 

The  MTCs  are  currently  coUecting  1990  and  1992  data  from  firms  that  they 
assisted  in  1991.  Results  of  client  progress  surveys  of  28  firms  assisted  by  the 
Great  Lakes  MTC  in  1991  are  illustrative.^^  For  example,  the  firms  reduced  their 


B-2-26 


manufacturing  lead  times  by  an  average  of  22  percent  and  reported  average 
increases  of: 

♦  13  percent  in  sales, 

♦  47  percent  in  value  of  exports, 

♦  6  percent  in  employment, 

♦  17  percent  in  total  payroll,  and 

♦  46  percent  in  the  use  of  computers  or  computer-controUed  equipment 
by  employees. 

To  increase  the  value  of  the  data  gathered  in  client  progress  surveys,  the  MEP  is 
develq)ing  a  nationwide  benchmariong  database  on  the  performance  of  up  to 
1,000  plants  to  help  companies  better  understand  their  competitive  position  in 
the  maifcetplace.  TTie  database  will  build  on  a  300-client  database  developed  by 
the  Midwest  MTC  in  Ann  Arboi;  Mich.,  which  now  contains  performance  data  for 
five  industries.  Hiis  will  enable  the  Midwest  MTC  to  provide  benchmarking  serv¬ 
ice  to  client  firms  of  other  extension  services.  In  addition,  the  eiqianded  database 
will  be  a  useful  MEP  management  tool,  enabling  national-level  comparisons  of 
the  performance  of  MEP  clients  with  the  performance  of  “control”  firms  that 
have  not  used  extension  services. 


Because  the  MEP  is  promoting  measurement  and  evaluation  as  powerful  learn¬ 
ing  and  management  tools,  it  has  ashed  the  Modernization  Fonim,  vdiich  draws 
its  Tn«^bpjship  from  the  centers,  to  develop  a  guide  on  eraluation  and  associated 
training  materials  for  use  by  project  managers  and  manufacturing  extension 
field  agents.  Training  workshops  are  now  being  plaimed,  with  the  first  scheduled 
for  Febmary  1994. 

Finally,  the  MEP  will  continue  to  sponsor  third-party  reviews  of  the  extension  net- 
woric  and  its  components  to  evaluate  their  effectiveness  and  to  guide  improve¬ 
ments  in  performance,  including  evaluation.  It  already  has  initiated  several 
efforts  to  build  a  common  understanding  of  process  and  outcome  evaluation.  For 
example,  it  has  sponsored  studies  of  evaluation  issues  and  practices  in  industrial 
modernization  programs,  one  conducted  by  the  Georgia  Institute  of  Technology 
and  the  other  by  Carnegie  Mellon  University.  It  also  requested  a  National  Re¬ 
search  Council  study  of  barriers  and  opportunities  to  improve  the  manufacturing 
performance  of  small  and  medium-si^  manufacturers.^^  Other  steps  taken  to 
build  a  national  framework  for  evaluation  include  the  continuing  development 
of  taxonomies  for  classifying  types  of  services,  activities,  and  manufacturing  prob¬ 
lems  and  practices  as  well  as  efforts  to  form  a  national  network  of  evaluators 
comprising  federal,  state,  local,  and  academic  eiqperts. 


The  MEP  is  promoting 
measurement  and 
evaluation  as 
powerful  learning  and 
management  tools. 
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Session  3 


Technology  Reinvestment  Project 


Technology,  the  key  to  America's  Cold  War 
military  strategy,  will  be  even  more  important 
in  the  unpredictable  security  environment  we 
now  face.  But  the  military  must  reduce  its 
reliance  on  a  "defense-unique"  technology 
and  industrial  base  in  response  to  two  new 
realities; 

•  Cold  War  defense  budgets  are  no  longer 
sustainable;  affordability,  along  with  per¬ 
formance,  must  now  be  a  primary  concern 
for  DoD. 

•  Many  of  the  technologies  most  critical  to 
defense  are  emerging  in  the  commercial 
sector;  currently,  defense  access  to  these 
technologies  is  limited. 

DoD's  dual-use  technology  strategy  is  a 
response  to  these  new  realities,  and  the  Tech¬ 
nology  Reinvestment  Project  (TRP)  is  DoD's 
largest  dual-use  technology  program. 

The  mission  of  the  TRP  is  to  give  the  Depart¬ 
ment  of  Defense  greater  access  to  affordable, 
leading-edge  technology  by  leveraging  com¬ 
mercial  know-how,  investments  and  markets 
for  military  benefit  This  mission  is  embodied 
in  two  strategic  goals:  (1)  to  enhance  the  tech¬ 
nological  superiority  of  defense  systems  while 
lowering  costs;  and  (2)  to  strengthen  the 
industrial  base  on  which  DoD  depends  while 
lowering  its  cost. 

To  achieve  these  goals,  the  TRP  awards  funds, 
on  a  cost-shared  basis,  to  industry-led  projects 
to  create  new  dual-use  technologies.  These 
technology  development  projects  are  of  two 
types,  corresponding  to  the  first  two  pillars  of 
DoD's  dual-use  technology  strategy. 

•  Leveraging  commercial  technology:  The 
first  type  of  project  advances  the  commer¬ 
cial  development  of  key  technologies  that 
meet  defense  needs.  Because  commercial 


demand  will  eventually  make  these  tech¬ 
nologies  more  affordable  to  the  military, 

DoD  benefits  by  accelerating  the  develop¬ 
ment  of  the  technology  while  simultane¬ 
ously  ensuring  that  it  meets  defense 
requirements. 

•  Transitioning  defense  technology:  The  sec¬ 
ond  type  of  project  promotes  the  transition¬ 
ing  of  defense  technologies  to  commercial 
applications.  The  creation  or  enhancement 
of  commercial  markets  for  these  technolo¬ 
gies  makes  them  more  affordable  and 
accessible  to  the  military. 

Although  the  lion's  share  of  funds  has  gone  to 
technology  development,  the  TRP  has  awarded 
smaller  amounts  of  money  to  projects  in  two 
other  areas:  technology  deployment,  to  build  a 
"dual-produce"  capability  in  the  U.S.  fnanufac- 
turing  base,  by  helping  small  defense  firms 
compete  in  commercial  markets;  and  manufac¬ 
turing  engineering  education,  to  reorient  engi¬ 
neering  education  to  the  dual-use  manufac¬ 
turing  industries  of  the  future.  Last  year,  the 
Department  of  Commerce  assumed  full 
responsibility  for  supporting  technology 
deployment. 

Winning  projects  are  chosen  solely  on  the 
basis  of  merit  by  technical  evaluators  from  the 
Department  of  Defense  and  other  federal  agen¬ 
cies.  To  ensure  that  TRP  projects  are  driven  by 
market  needs,  participants  must  contribute  at 
least  half  the  cost  of  the  project. 

The  TRP  is  implemented  by  the  Department  of 
Defense  (ARPA  and  the  military  services), 
working  jointly  with  five  other  agencies:  the 
Departments  of  Commerce,  Energy  and  Trans¬ 
portation,  the  National  Science  Foundation  and 
the  National  Aeronautics  and  Space  Adminis¬ 
tration.  The  FY93  budget  for  the  TRP  was  $472 
million;  in  FY94  the  program  received  $404 
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million.  The  FY95  appropriation  is  $443  million. 
The  Presidejifs  FY96  Budget  requests  $500  mil¬ 
lion  for  the  TRP. 

President  Clinton  unveiled  the  TRP  on  March 
11,  1993,  The  subsequent  competition  was 
heavily  oversubscribed:  the  TRP  received  2,850 
proposals,  requesting  $8.5  billion.®  These  pro¬ 
posals  were  evaluated  by  300  experts  from 
DoD  and  other  federal  agencies.  Between 
October  1993  and  February  1994,  the  TRP 
announced  awards  of  matching  federal  funds 
totaling  $605  million  to  212  projects,  involving 
1,600  firms,  universities  and  other  participants. 
Awards  went  to  all  of  the  1993  proposals  that 
were  "highly  recommended". 

In  March  1994,  the  TRP  announced  1993  Small 
Business  Innovation  Research  (SBIR)  awards  of 
$15  million  to  153  small  businesses. 

A  second  TRP  competition  got  underway  in 
April  1994.  In  October,  the  TRP  awarded  $200 
million  in  federal  matching  funds,  bringing  to 
$820  million  the  total  size  of  awards 


announced  so  far  under  the  TRP.  The  awards 
went  to  39  projects  involving  224  participants. 

A  third  TRP  competition,  to  allocate  $415  mil¬ 
lion,  was  announced  on  October  21,  1994.  DoD 
will  announce  the  awards  in  mid-1995. . 

Assessing  the  TRP:  Insights  from  the 
First  Two  Years 

Few  federal  programs  have  been  as  enthusi¬ 
astically  received  and  as  carefully  scrutinized 
as  the  TRP.  The  program  has  received  consid¬ 
erable  praise  as  well  as  its  share  of  criticism — 
much  of  that  the  result  of  a  misunderstanding 
of  its  mission  (see  Box  G).  In  response  to  sug¬ 
gestions  from  industry,  ARPA  has  modified  the 
program  to  increase  applicants'  success  rate 
and  expand  participation  by  smalt  business.  In 
addition,  the  TRP  has  increased  the  military 
services'  involvement  in  the  program,  to 
ensure  that  TRP-developed  technologies  are 
rapidly  integrated  into  defense  weapon  systems. 


^Industry  responded  aggressively  to  the  TRP  solicitation  despite  its  complexity  (eight  statutory  programs,  each  with  its  own  require¬ 
ments);  the  $8.5  billion  in  requested  funds  exceeded  available  funds  by  a  factor  of  17  overall,  and  by  as  much  as  30  or  40  for  some  of  the 
technology  development  programs.  Moreover,  many  of  the  proposals  were  very  good:  Although  the  TRP  funded  all  of  the  "highly  recom¬ 
mended  proposals  about  7  percent — the  program  was  unable  to  fund  any  of  a  large  number  of  proposals  that  were  "recommended" 
by  evaluators. 


BOXG 

TRP  Mission;  Dual  Use  Not  "Defense  Conversion" 

Although  the  Technology  Reinvestment  Project  has  completed  two  competition 
cycles  and  awarded  more  than  $800  million  in  matching  funds,  its  basic  mission  is 
still  not  fully  understood.  A  "defense  conversion"  program  has  the  aim  of  convert¬ 
ing  defense  firms  and  workers  from  military  to  civilian  production;  that  is  nof  the  pur¬ 
pose  of  TRP,  Rather,  the  TRP  mission  is  to  increase  defense  access  to  affordable,  lead¬ 
ing-edge  technology  by  leveraging  commercial  capabilities  and  markets.  The  pro¬ 
gram  pursues  this  mission  in  two  ways,  corresponding  to  Pillars  1  and  2  of  the 
Department  of  Defense  dual-use  technology  plan.  First,  it  accelerates  the  develop¬ 
ment  of  emerging  commercial  technologies,  such  as  flat  panel  displays  and  high- 
density  data  storage,  that  are  critical  to  defense.  Second,  the  TRP  helps  transition 
defense  technologies  to  commercial  applications  where  that  will  benefit  DoD. 

The  second  approach  may  be  confused  with  "defense  conversion,"  but  the  differ¬ 
ences  are  key.  The  TRP's  customer  is  the  Pentagon:  The  program  supports  "spinoff" 
of  defense  technology  only  if  it  preserves  access  to  a  technology  or  makes  it  more 
affordable  to  the  military.  Moreover,  the  TRP,  in  effect,  requires  defense  firms  to  part¬ 
ner  with  non-defense  firms  that  have  know-how  in  marketing  and  low-cost  produc¬ 
tion.  Experience  has  shown  that  most  defense  firms  cannot  "convert"  their  high- 
overhead,  cost-plus  culture  to  compete  in  commercial  markets.  Thus  the  most  effec¬ 
tive  way  to  move  defense  technology  Into  commercial  markets  is  through  partner¬ 
ships  with  non-defense  firms. 

Some  have  questioned  the  defense  relevance  of  TRP  projects  In  health  care,  the  envi¬ 
ronment  and  other  seemingly  nontraditional  areas.  In  fact,  medical  technology  has 
long  been  a  mission  interest  of  the  military  services — e.g.,  the  Army  and  Navy's 
research  in  telemedicine.  And  environmental  technology  meets  two  important 
defense  needs:  detection  of  biological/chemical  warfare  agents,  and  monitoring  and 
cleanup  of  contaminants  at  DoD  sites.  More  generally,  the  process  for  selecting  TRP 
focus  areas,  evaluating  proposals  and  managing  projects  ensures  that  the  program 
serves  military  needs  foremost 

Misunderstandings  about  the  program  have  led  some  to  characterize  the  TRP  and 
other  dual-use  technology  programs  as  a  "nontraditional"  defense  expenditure  -  that 
Is,  an  expenditure  that  does  not  contribute  to  traditional  DoD  military  activities.  That 
label  is  incorrect  for  two  reasons. 

First,  for  50  years  technology  has  been  the  basis  of  this  country's  military  advantage, 
and  defense  R&D  has  been  the  key  to  that.  The  TRP  and  other  DoD  dual-use  tech¬ 
nology  programs  represent  a  different — and  more  effective — way  of  carrying  out  cer¬ 
tain  critical  defense  R&D  activities. 

Second,  dual  use  itself  has  a  rich  tradition  in  DoD,  particularly  ARPA,  where  program 
managers  in  the  1950s  and  1960s  consciously  promoted  industrial  spinoffs  as  a  way 
to  lower  the  cost  of  technology  to  the  military.  That  visionary  approach  later  fell  vic¬ 
tim  to  budgetary  pressures  as  well  as  political  pressures  to  limit  the  Pentagon's 
impact  on  society  (the  word  "Defense"  was  added  to  ARPA's  name  in  1972,  as  a  way 
to  restrict  the  agency).  The  Clinton  Administration's  support  for  expanded  dual-use 
R&D — like  the  reinstatement  of  ARPA's  original  name — marks  a  return  to  the  agency's 
traditional  approach.* 


^his  history  is  described  in  a  1992  article  by  Admiral  Bobby  R.  Inman  and  Sen.  Jeff  Bingaman  :hat  urges  Congress,  for  the  sake  of 
national  security,  to  "restore  the  Pentagon's  tradittonal  role  of  promoting  technology  with  commercial  as  well  as  military  uses."  "Broadening 
Horizons  for  Defense  R&D."  Issues  in  Science  and  Technology  (Fall  1992), 


B-3-3 


Second  to  None:  Preserving  America's  Military  Advantage  Through  Dual-Use  Technology 


One  indication  of  the  TRP's  success  is  the 
amount  of  private  investment  it  has  catalyzed. 
In  1993,  winning  projects  matched  every  feder- 
ai  dollar  with  $1,40  of  non*federal  funds — $845 
million.  The  cost  share  by  1994  winners  was 
also  significantly  above  the  dollar-for-dollar 
match  required  by  the  TRP.  In  all,  the  federal 
TRP  investment  of  $820  million  is  leveraging 
projects  worth  $1.9  billion.  By  putting  private 
sector  money  at  risk,  the  cost  share  also 
ensures  industry's  commitment  to  the  project 
and  lays  the  foundation  for  industry  to  assume 
the  total  cost  of  commercialization. 

To  provide  more  systematic  feedback,  ARPA 
and  the  interagency  TRP  Working  Group  are 
developing  a  formal  structure  to  monitor  and 
assess  the  impact  of  the  program  in  terms  of 
its  strategic  goals.  Although  it  is  too  early  to 
judge  the  technical  progress  of  many  TRP  pro¬ 
jects,  much  less  their  final  outcome,  an  analy¬ 
sis  of  the  proposals  and  awards  provides  a 
preliminary  picture  of  the  program's  direction 
and  impact. 

Significant  Benefits  to  Defense 
Systems 

The  TRP  will  yield  direct  benefits  to  the  mili¬ 
tary  in  the  form  of  technologies  and  products 
that  the  military  services  can  use.  Indirect  ben¬ 
efits  include  cost  savings,  advanced  commer¬ 
cial  capabilities  for  military  use,  and  preserva¬ 
tion  of  the  defense  industrial  base.  (See  Chart 
1  for  sample  metrics.) 

The  TRP  has  been  carefully  designed  to  ensure 
that  military  needs  are  given  priority  consider¬ 
ation:  DoD  representatives  oversee  program 
planning,  and  the  military  services  are  closely 
involved  in  ail  aspects  of  program  execution. 

For  example,  for  the  1993  technology  develop¬ 
ment  competition: 

•  over  one-third  of  the  TRP  evaluators  were 
from  DoD; 

•  the  military  services,  service  laboratories 
and  defense  agencies  are  managing  two- 
thirds  of  TRP  funds; 

•  service  laboratories  and  other  DoD  facilities 
are  participants  in  30  percent  of  the  projects. 

The  most  persuasive  evidence  of  the  military 
benefit  is  the  projects  themselves:  All  TRP  pro¬ 


jects  meet  clear  defense  needs,  as  well  as  con¬ 
tributing  to  the  broader  industrial  base.  Box  H 
(below)  describes  some  typical  TRP  projects; 
they  are  developing  dual-use  technologies  to 
provide  for  affordable  night  vision  capability, 
to  improve  battlefield  casualty  treatment,  and 
to  make  affordable  the  Army's  technically 
superior,  but  high-cost,  system  for  locating 
combat  units  on  the  battlefield  in  real  time.  By 
taking  advantage  of  the  potential  for  a  com¬ 
mercial  market,  these  projects  offer  the 
prospect  of  technology  with  improved  perfor¬ 
mance  at  lower  cost  to  DoD. 

As  an  overview.  Chart  2  shows  the  number 
and  value  of  1993  technology  development 
awards  in  each  technology  category.  (These 
technology  categories  are  more  broadly 
defined  than  the  focused  technology  areas  in 
the  April  1994  competition.)  Three  of  the  1993 
technology  development  categories  contribute 
directly  to  the  performance  and  affordability  of 
DoD  weapon  system  platforms:  vehicle  tech¬ 
nologies  (including  advanced  batteries),  ship¬ 
building,  and  aerospace.  These  three  areas 
received  about  30  percent  of  available  funds. 
(Chart  3.) 

For  example,  Martin  Marietta  leads  an  effort  to 
develop  a  new  airport  radar  system  that  can 
detect  hazardous  weather  conditions  while 
simultaneously  monitoring  air  traffic,  using  a 
technology  first  developed  for  the  Navy.  Com¬ 
mercialization  of  this  technology  for  the  civil¬ 
ian  aviation  market  will  yield  better  and  cheap¬ 
er  technology  to  meet  military  needs  in  tactical 
ballistic  missile  defenses  and  airborne  fire-con¬ 
trol  systems.  To  take  another  example,  a 
Miami-based  team  led  by  Pratt  &  Whitney  is 
developing  lightweight,  polymer  composites 
for  aircraft  engines.  Advanced  composites  will 
significantly  increase  the  performance  and 
range  of  military  aircraft,  while  lowering  the 
cost  of  repair  and  maintenance. 

Dual-use  process  improvements — in  electronic 
design  and  manufacturing,  mechanical  design 
and  manufacturing,  and  materials  and  struc¬ 
tures  manufacturing — promise  significant 
improvements  in  the  cost  of  producing 
defense  systems.  (Chart  4.)  For  example,  the 
Precision  Laser  Machining  Project  (Box  E, 
above)  will  develop  a  new  class  of  high-speed, 
high-precision  laser  machine  tools  with  wide¬ 
spread  application  to  military  as  well  as  com¬ 
mercial  production.  Among  other  things,  these 
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new  laser  tools  will  make  it  possible  to 
increase  aircraft  fuel  savings  by  several  per¬ 
cent,  by  improving  the  uniformity  of  millions 
of  microscopic  holes  drilled  in  airframes  to 
reduce  wind  drag. 

Information  infrastructure  projects,  which 
received  30  percent  of  1993  technology  devel¬ 
opment  funds,  directly  enhance  defense  elec¬ 
tronics  and  communications  systems.  (Chart 
5.)  The  Gulf  War  provided  a  glimpse  of  the  rev¬ 
olutionary  potential  of  these  technologies, 
which  can  boost  dramatically  the  range  and 
accuracy  of  conventional  weapons  such  as 
bombs  and  missiles,  and  can  design  the  next 
generation  of  aircraft,  ships  and  missiles.  In 
addition  to  reducing  the  cost  to  DoD  of  infor¬ 
mation  technology  through  dual-use  applica¬ 
tions,  projects  in  this  category  will  improve 
fiber  optic  transmission,  signal  processing, 
radar  imaging,  wireless  communications,  and 
radar  frequency  modules.  Other  projects  will 
create  new  dual-use  capabilities  such  as 
speech-activated  hand-held  computers,  soft¬ 
ware  standards,  and  advanced  techniques  for 
manufacturing  efficiency. 

So  important  is  information  technology  to  the 
military  that  the  April  1994  TRP  competition 
targeted  five  focused  technologies  in  the  infor¬ 
mation  and  electronics  area.  (Chart  6.)  One 
"focus  area"  is  high-density  data  storage 
devices.  Because  commercial  needs  will  even¬ 
tually  drive  the  market,  DoD  benefits  by  stimu¬ 
lating  and  accelerating  that  market.  Vast 
increases  in  portable,  low-cost  data  storage 
will  allow  DoD  to  take  full  advantage  of  the 
growing  availability  of  high  resolution  map¬ 
ping  images  during  military  operations,  and 
will  give  our  front-line  soldiers  immediate 
access  to  the  best  information  and  intelligence. 
Another  group  of  projects  will  help  ensure  the 
creation  of  open,  interoperability  standards  for 
the  National  Information  Infrastructure  in  areas 
that  are  defense-critical.  High-definition  sys¬ 
tems  manufacturing  technology  is  a  third 
focus  area:  These  systems  (also  known  as  flat 
panel  displays)  will  be  as  important  to  Ameri¬ 
can  soldiers  in  future  conflicts  as  two-way 
radios  and  GPS  receivers  were  in  Desert  Storm. 

The  1994  competition  also  targeted  two  sensor 
technologies.  (Chart  7.)  Uncooled  infrared  sen¬ 
sors,  to  take  one,  offer  a  potentially  affordable 


approach  to  providing  night  vision  capability 
to  combat  troops;  the  widespread  use  of  effec¬ 
tive  infrared  devices  on  the  battlefield  could 
revolutionize  our  ability  to  fight  under  cover  of 
night,  fog  or  smoke.  The  devices  now 
employed  by  the  military  require  cryogenic 
cooling,  whose  cost  is  prohibitive  for  wide¬ 
spread  use  by  troops.  Commercial  develop¬ 
ment  of  uncooled  sensors — e.g.,  for  police 
work  and  enhancement  of  vision  in  cars  and 
trucks — could  bring  improved  performance 
and  a  tenfold  cost  reduction  for  military  users 
(see  Box  H,  below). 

Summary:  Defense  Benefits  of  "Spinoff" 
and  "Spin-on" 

Like  uncooled  infrared  sensors,  about  half  of 
all  TRP  projects  serve  defense  needs  by  mov¬ 
ing  DoD-funded  technologies  into  commercial 
applications.  Despite  the  size  and  sophistica¬ 
tion  of  the  defense  technology  base,  the  bulk 
of  defense  R&D  never  leaves  the  defense  sec¬ 
tor  to  build  commercial  capabilities.  TRP  pro¬ 
jects,  as  Chart  8  shows,  are  replete  with  new 
uses  for  defense  technologies,  from  the  appli¬ 
cation  of  amorphous  silicon  to  medical  imag¬ 
ing  to  the  use  of  advanced  composites  for 
bridge  repair.  DoD  benefits  primarily  from  the 
lower  costs  achieved  through  more  efficient 
production  and  economies  of  scale.  In  addi¬ 
tion,  some  TRP  projects  contribute  to  more 
than  affordability.  For  example,  the  work  of  the 
Precision  Laser  Machining  Project  will  yield 
back  a  superior  technique  for  jamming  the 
sensors  of  heatseeking  missiles. 

Chart  9  summarizes  the  defense  benefits  from 
several  TRP  projects  that  "spin-on"  emerging 
commercial  technologies.  Defense  often  trails 
in  these  technologies;  thus  the  projects  pro¬ 
vide  the  military  with  superior  technology  that 
will,  over  time,  become  affordable  because  of 
the  potential  for  a  self-sustaining  commercial 
industry.  Flat  panel  displays  and  high-density 
data  storage  devices  fit  this  pattern.  Another 
example  is  technology  for  treatment  of  battle¬ 
field  casualties,  including  digital  X-Rays,  sen¬ 
sors,  and  information  technology.  In  one  TRP 
project,  General  Electric  has  teamed  with 
EG&G  in  a  two-year  program  to  develop  a  Dig¬ 
ital  X-Ray  System  for  Trauma  and  Battlefield 
Applications  (see  Box  H,  below). 
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Improvements  to  Industrial  Base 
Supporting  Defense 

In  addition  to  providing  direct  defense  bene¬ 
fits,  TRP  projects  strengthen  the  industrial 
base  supporting  defense.  The  result  of  these 
investments  will  be  a  stronger  and  more 
diverse  industrial  base  capable  of  making 
more  affordable  products  for  the  military. 

(Chart  10.) 

One  way  the  TRP  strengthens  the  industrial 
base  for  defense  is  by  expanding  the  market 
for  defense-dependent  firms  that  cannot  sus¬ 
tain  themselves  by  military  sales  alone.  Many 
of  these  firms  have  technologies  and  core 
competencies  that  have  application  in  the 
commercial  sector.  Under  the  TRP,  these  firms 
are  teamed  with  commercial  firms  that  under¬ 
stand  low-cost  production  and  marketing,  to 
develop  viable  "spinoffs"  for  new  markets. 

Similarly,  the  TRP  is  helping  to  preserve 
defense-unique  capabilities  that  might  other¬ 
wise  disappear.  One  example  is  Hi-Shear 
Corp.,  a  small  Torrance,  California,  firm  that 
makes  military  detonators.  With  the  help  of  a 
TRP  award,  Hi-Shear,  in  partnership  with  the 
Torrance  Fire  Department,  is  adapting  its 
pyrotechnic  technology  for  use  in  emergency 
rescue  equipment.  By  developing  new,  civilian 
markets  for  its  technology,  Hi-Shear  will  be 
able  to  remain  a  defense  supplier. 

Third,  a  number  of  TRP  projects  are  spinning 
off  defense  technologies  that  have  application 
to  the  manufacturing  process  itself,  which 
enhances  the  industrial  base  for  defense  as 
well  as  commercial  production.  (Chart  11.)  For 
example,  DoD's  3D  printing  capability  will  be 
used  to  increase  the  precision  of  injection 
molding  for  ceramics,  which  will  reduce  the 
cost  of  components  for  commercial  and  mili¬ 
tary  jet  engines;  information-related  technolo¬ 
gies  will  contribute  to  more  precise  computer- 
controlled  manufacturing  processes  and  the 
"paperless  factory."  In  many  cases,  the  origi¬ 
nal  developer  of  the  technology  was  aware  of 
the  commercial  potential  but  lacked  the  com¬ 
mercial  expertise  and  financial  support  that  the 
TRP  partnering  process  provides. 

The  TRP  will  be  measured  in  considerable  part 
by  its  contribution  to  breaking  down  the  barri¬ 
ers  that  separate  the  civilian  and  defense  sec¬ 
tors.  Judging  from  its  first  two  years,  the  pro¬ 
gram  is  succeeding.  By  design,  the  TRP  has 


attracted  a  broad  mix  of  public  and  private 
participants  and  stimulated  an  enormous 
amount  of  collaboration  and  alliance-building, 
both  within  industry,  and  between  industry, 
universities  and  federal  laboratories.  Many 
losing  teams  confided  that  they  had  gained 
from  the  process,  and  a  number  of  teams  said 
they  would  proceed  without  federal  funds 
(albeit  on  a  smaller  scale  or  without  as  much 
payoff  to  DoD). 

Most  notably,  TRP  technology  development 
projects  bring  together  teams  that  are  well- 
integrated,  both  horizontally  and  vertically:  A 
typical  winning  team  includes  a  large  defense 
prime,  a  large  commercial  prime,  one  or  more 
small  firms,  and  a  university  or  federal  lab. 

The  presence  of  commercial  firms  on  most 
teams  Is  particularly  important,  because  it 
expands  the  industrial  base  serving  DoD  and 
ensures  that  the  resulting  technology  will  be 
commercially  viable.  In  fact,  the  April  1994  TRP 
competition,  with  its  emphasis  on  leading- 
edge  electronics  technology,  produced  a  num¬ 
ber  of  awards  to  teams  led  by  commercial  firms. 

In  addition,  TRP  teams  benefit  from  strong 
participation  by  public  and  non-profit  organiza¬ 
tions,  Including  universities,  medical  institu¬ 
tions,  public  and  private  laboratories,  and 
government  entities  at  ail  levels.  (Among  suc¬ 
cessful  technology  development  projects,  one- 
third  of  participants  are  from  outside  of  indus¬ 
try.)  Universities  and  labs  are  important 
sources  of  advanced  technology  with  potential 
military  application.  Government  participants 
often  provide  the  perspective  of  a  non-DoD 
customer  for  dual-use  technology — as,  for 
example,  in  the  development  of  equipment  for 
toxic  waste  clean-up  or  advanced  firefighting 
services. 

The  TRP  also  produced  an  unprecedented  part¬ 
nership  among  agencies  within  the  federal 
government.  DoD  got  the  benefit  of  the  other 
agencies'  expertise  In  civilian  technology  appli¬ 
cations,  while  the  non-defense  agencies  got 
valuable  exposure  to  military  technologies 
with  potential  relevance  to  their  mission 
needs.  For  example,  the  Department  of  Trans¬ 
portation  is  now  looking  at  increasing  its  fund¬ 
ing  to  adapt  advanced  composites  for  con¬ 
struction  of  bridges  and  roads,  which  will  help 
to  preserve  the  composites  industrial  base  and 
bring  down  the  cost  of  those  materials  for  mili¬ 
tary  aircraft.  Cooperation  among  the  agencies 
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has  increased  outside  of  the  TRP,  as  a  result  of 
their  greater  awareness  of  one  another's  activi¬ 
ties  in  dual-use  technology.  Of  lasting  value, 
the  civilian  agencies  have  begun  to  adopt 
ARPA's  more  streamlined  approach  to  funding 
R&D,  including  greater  flexibility  toward  intel¬ 
lectual  property  rights. 

Lessons  Learned.  The  TRP's  1993  competi¬ 
tion  revealed  two  key  areas  for  improvement: 
The  success  rate  of  proposals  was  too  low, 
resulting  in  wasted  bid  and  proposal  expendi¬ 
tures  by  unsuccessful  applicants.  And  partici¬ 
pation  by  small  firms  was  too  limited. 

To  raise  the  success  rate  for  TRP  applicants, 
the  1994  competition  targeted  specific  technol¬ 
ogy  focus  areas  that  are  of  direct  military  inter¬ 
est.  Among  other  advantages,  this  targeting 
gave  prospective  applicants  a  better  basis  for 
deciding  whether  to  apply.  Second,  the  TRP 
conducted  more  outreach  to  prospective  appli¬ 
cants,  including  day-long  workshops  on  each 
technology  focus  area.  Third,  prospective 
teams  were  encouraged  to  submit  "white 
papers"  for  review,  prior  to  preparing  a  full 
proposal.  As  a  result,  although  requested 
funds  still  exceeded  available  funds  by  a  factor 
of  four  overall,  the  success  rate  of  technology 
development  proposals  was  18  percent,  which 
is  more  typical  of  other  ARPA  programs. 

To  Improve  the  ability  of  small  companies  to 
participate,  the  TRP  took  advantage  of  new 
legislation  permitting  small  firms  to  use  SBIR 
grants  as  part  of  their  cost  share.  Small  firms 
also  were  given  up  to  120  days  following  the 
announcement  of  an  award  to  come  up  with 
their  share  of  project  costs.  These  changes, 
combined  with  a  more  aggressive  outreach 
effort  to  small  firms,  resulted  in  a  higher  par¬ 
ticipation  rate:  70  percent  of  the  1994  develop¬ 
ment  teams  included  one  or  more  small  firms, 
compared  to  50  percent  in  1993.'° 

In  addition  to  these  changes,  the  TRP  has 
increased  the  level  of  involvement  by  the  mili¬ 
tary  services  even  more,  to  ensure  that  tech¬ 
nologies  developed  with  program  funds  are 
rapidly  integrated  into  defense  systems.  The 
services  played  a  larger  role  in  the  selection  of 
focus  areas  for  the  1995  competition,  and  they 
are  now  represented  formally  on  the  Defense 


Technology  Coordinating  Council,  which  over¬ 
sees  the  TRP 

1995  Competition.  The  Clinton  Administra¬ 
tion  is  committed  to  continuing  and  to  improv¬ 
ing  the  TRP.  Program  staff  have  conducted 
workshops  around  the  country  both  to  provide 
feedback  to  unsuccessful  applicants  and  to 
solicit  ideas  for  improving  the  program.  The 
TRP  held  several  two-day  workshops  to  help 
participants  form  and  maintain  R&D  partnerships. 

The  1995  competition,  announced  on  October 
21,  1994,  will  allocate  $415  million  in  matching 
funds;  the  bulk  of  the  money  will  go  to  tech¬ 
nology  development  projects  in  12  dual-use 
focus  areas  selected  for  their  defense  relevance: 

•  Digital  wireless  communications  and  net¬ 
working  systems — Development  of  inno¬ 
vative  communications  and  networking 
products  to  promote  the  "digitization  of  the 
battlefield." 

•  Affordable  polymer  matrix  composites  for 
airframe  structures — Development  of  mate¬ 
rials  and  manufacturing  technologies  for 
affordable  fabrication  of  primary  airframe 
composite  structures  to  improve  military 
and  commercial  aircraft  performance  and 
cost-effectiveness. 

•  Microelectromechanical  systems  (MEMS) 
applications — Demonstration  and  insertion 
of  MEMS  technology  into  defense  and  com¬ 
mercial  applications  (inertial  sensors, 
embedded  detection  devices,  etc.). 

•  Low-cost  specialty  metals  processing — 
Demonstration  and  insertion  of  innovative 
forming  of  component  fabrication  processes 
to  make  specialty  metals  more  affordable 
for  military  and  commercial  use. 

•  Millimeter  wave  products — Development  of 
affordable  and  reliable  millimeter  wave 
products  using  monolithic-format  Integrated 
circuits  for  use  in  military  and  commercial 
applications. 

•  Electric  and  hybrid  tactical  and  commercial 
vehicles — Development  of  affordable  medi- 
um-to-heavy  hybrid  electric  drivetrains  for 
military  and  commercial  use. 


’"Overall,  small  firms  have  done  best  in  the  TRP  competition  as  participants  rather  than  project  sponsors.  This  reflects  DoD's  strong  pref¬ 
erence  for  project  teams  that  are  committed  to  commercialization:  Many  small  technology-oriented  firms  lack  commercialization  and  produc¬ 
tion  know-how.  and  TRP  proposals  led  by  small  firms  often  have  not  included  other  firms  with  compensating  expertise. 
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Chart  1 

Sample  Metrics  -  Defense  Benefits  from  TRP 
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Technical  Progress  and  Commercialization 

•  On  time  progress  toward  technical  milestones  for  defense  applications 

•  Transfer  of  technology  and  expertise  within  team 

•  Identification  and  coordination  with  potential  defense  customers 

•  Progress  toward  commercializing  product  for  DoD  market 


Outcome  -  Analysis  of  Results 

•  Breadth  and  criticality  of  military  applications 

•  Cost  savings  achieved  through  dual-use  approaches 

•  Increases  in  DoD  use  of  commercial  technologies  and  items 

•  Impact  of  "spinoffs"  on  defense  costs 

•  Preservation  of  defense  industrial  base,  skills,  processes  and  facilities 
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Chart  10 

Sample  Metrics  -  Industrial  Base  Benefits  from  TRP 
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Technical  Progress  and  Commercialization 

•  Effective  coordination  of  responsibilities  among  partners 

•  Progress  toward  product  that  responds  to  diverse  market  needs 

•  Transfer  of  knowledge  and  expertise  between  participants 

•  Progress  toward  commercialization 


Outcome  -  Analysis  of  Results 

•  Increase  in  diversity  within  defense  supplier  base 

•  Transition  to  integrated  facilities 

•  Expansion  of  market/improved  viability  of  defense-dependent  firms 

•  Increase  in  use  of  dual-use  and/or  commercial  products  by  DoD 

•  Expansion  of  privately  funded  TRP-like  activity 
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ECONOMIC  METRICS  FOR  TECHNOLOGY: 
A  SELECTED  BIBLIOGRAPHY 


Economic  Metrics  for  Technology: 
A  Selected  Bibliography 


I.  Economic  Criteria  for  Investment  and  Evaluation 

a.  Ex-Ante  Criteria  for  Technology  Investment 

Examples : 

Government 

Advisory  Council  on  Federal  Participation  in  SEMATECH.  SEMATECH 
1990:  A  Report  to  Congress.  Washington,  D.C.:  May  1990. 

Branscomb,  Lewis  M.,  ed.  Empowering  Technology;  Implementing  a 
U.S.  Strategy.  Cambridge:  The  MIT  Press,  1993. 

Clark,  Marianne  K.  and  Dobson,  Eric.  Promoting  Technical 
Excellence:  The  Role  of  State  and  Federal  Extension 

Activities .  Washington,  D.C.:  National  Governor's 
Association,  1989. 

Clinton,  William  J.,  and  Gore,  Albert.  "Technology  for  America's 
Economic  Growth:  A  New  Direction  to  Build  Economic  Strength," 
February  1993. 

COSEPUP.  "The  Outlook  for  Science  and  Technology:  The  Next  Five 
Years,"  a  Report  by  the  Committee  on  Science,  Engineering  and 
Public  Policy  for  the  NSF,  Washington,  D.C.:  National  Academy 
Press,  1982. 

COSEPUP.  "Research  Briefings,"  prepared  for  the  Office  of  Science 
and  Technology  Policy,  the  National  Science  Foundation,  and 
selected  federal  departments  and  agencies  by  the  Committee  on 
Science,  Engineering  and  Public  Policy,  Washington,  D.C.:  U.S. 
Government  Printing  Office,  1983. 

Leyden,  D.  and  Link,  A.  Government's  Role  in  Consortia.  Boston: 
Kluwer  Academic  Publishers,  1992. 

Office  of  Technology  Assessment.  Federally  Funded  Research: 
Decisions  for  a  Decade.  May,  1991. 

Industry 

Evan,  W.  and  Oik,  P.  "R&D  Consortia:  A  New  U.S.  Organizational 
Form,"  Sloan  Managaement  Review.  Spring  1990,  pp.  37-46. 

Grindley,  Peter;  Mowery,  David  C.  and  Brian  Silverman.  "SEMATECH 
and  Collaborative  Research:  Lessons  in  the  Design  of  High 
Technology  Consortia, "  Journal  of  Policy  Analysis  and 
Management ,  vol.  13,  no.  4,  1994,  pp.  723-758. 

Critiques : 

Government 

Cohen,  Linda  and  Noll,  Roger.  The  Technology  Pork  Barrel . 
Washington,  D.C.:  The  Brookings  Institution,  1991. 
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Gibbons,  M.  and  Georghiou,  L.  Evaluation  of  Research;  A  Selection 
of  Current  Practices .  Paris:  OECD,  1987,  pp.  59-64. 


Irvine,  John  and  Martin,  Ben.  Foresight  in  Science,  Picking  the 
Winners .  London:  Frances  Pinter,  1984,  chapter  5. 

Mowery,  David.  "Economic  Theory  and  Government  Technology  Policy," 
Policy  Sciences.  1983,  pp.  27-43. 

Peer  Review 

Bozeman,  Barry.  "Peer  Review  and  Evaluation  of  R&D  Impacts,"  in 
Evaluating  R&D  Impacts:  Methods  and  Practice,  eds . :  Barry 
Bozeman  and  Julia  Melkers.  Boston:  Kluwer  Academic 
Publishers,  1993. 

Chubin,  Daryl  E.  "Grants  Peer  Review  in  Theory  and  Practice," 
Evaluation  Review,  vol.  18,  no.  1,  February  1994,  pp.  20-30. 

Gibbons,  M.  and  Georghiou,  L.  Evaluation  of  Research:  A  Selection 
of  Current  Practices.  Paris:  OECD,  1987,  pp.  26-58. 

Kostoff,  Ronald  N.  "Assessing  Research  Impact:  Federal  Peer  Review 
Practices,"  Evaluation  Review,  vol.  18,  no.  1,  February  1994, 
pp.  31-40. 

National  Institutes  of  Health  (NIH) .  "Sustaining  the  Quality  of 
Peer  Review:  A  Report  of  the  Ad-Hoc  Panel,"  Bethesda,  Md. : 
NIH,  1989. 

b.  Ex-Post  Methods  of  Evaluation 

Examples : 

Industry 

Cooper,  Ronald  S.  "Assessing  Manufacturing  Modernization:  An 
Evaluation  of  our  Information  Sources, "  Foundation  for 
Industrial  Modernization,  December  1993. 

Enos,  John.  "Invention  and  Innovation  in  the  Petroleum  Refining 
Industry, "  in  The  Rate  and  Direction  of  Inventive  Activity. 
ed:  Richard  Nelson.  Princeton:  Princeton  University  Press, 
1962,  pp.  299-322. 

Katz,  Barbara  Goody,  and  Phillips,  Almarin.  "The  Computer 
Industry,"  in  Government  and  Technical  Progress,  ed. :  Richard 
Nelson.  New  York:  Pergamon  Press,  1982,  pp.  162-232. 

Levin,  Richard.  "The  Semiconductor  Industry,"  in  Government  and 
Technical  Progress,  ed. :  Richard  Nelson.  New  York:  Pergamon 
Press,  1982,  pp.  9-100. 

Mowery,  David  and  Rosenberg,  Nathan.  "The  Commercial  Aircraft 
Industry,"  in  Government  and  Technical  Progress,  ed. :  Richard 
Nelson.  New  York:  Pergamon  Press,  1982,  pp.  101-161. 
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Mowery,  David  and  Rosenberg,  Nathan.  Technology  and  the  Pursuit  of 
Economic  Growth.  Cambridge:  Cambridge  University  Press,  1989. 


Mueller,  Willard  F.  "The  Origins  of  the  Basic  Inventions 
Underlying  Dupont's  Major  Product  and  Process  Innovations, 
1920-1950,"  in  The  Rate  and  Direction  of  Inventive  Activity, 
ed. :  Richard  Nelson.  Princeton:  Princeton  University  Press, 
1962,  pp.  323-358. 

Peck,  Merton  J.  "Inventions  in  the  Post-War  American  Aluminum 
Industry, "  in  The  Rate  and  Direction  of  Inventive  Activity, 
ed:  Richard  Nelson.  Princeton:  Princeton  University  Press, 
1962,  pp.  279-298. 

Robb,  Walter  L.  "Evaluating  Industrial  R&D, "  Evaluation  Review, 
vol.  18,  no.  1,  February  1994,  pp.  89-97. 

Rosenberg,  Nathan.  "Technological  Change  in  the  Machine  Tool 
Industry,  1840-1910,"  Journal  of  Economic  History,  vol.  23, 
pp.  414-443. 

Econometric  Analysis 

Bernstein,  Jeffery  I.  and  Nadiri,  M.  Ishaq.  "Interindustry  R&D 
Spillovers,  Rates  of  Return,  and  Production  in  High-Tech 
Industries,"  American  Economic  Review  Papers  and  Proceedings, 
vol.  78,  no.  2,  May  1988,  pp.  429-434. 

Bresnahan,  Timothy.  "Measuring  the  Spillovers  from  Technical 
Advance,"  American  Economic  Review,  vol.  76,  no.  4,  September 
1986,  pp.  741-755. 

Griliches,  Zvi.  "Hybrid  Corn:  An  Exploration  in  the  Economics  of 
Technological  Change."  Econometrica.  vol.  25,  no.  4,  1957,  pp. 
501-522. 

Griliches,  Zvi.  "Research  Expenditures,  Education  and  the 

Agricultural  Production  Function, "  American  Economic  Review, 
vol.  54,  no.  6,  December  1964,  pp.  961-974. 

Griliches,  Zvi.  "Returns  to  Research  and  Development  in  the 

Private  Sector, "  in  New  Developments  in  Productivity 
Measurement  and  Analysis,  eds.:  J.  Kendrick  and  B.  Vaccara. 
Chicago:  University  of  Chicago  Press,  1984,  pp.  419-454. 

Griliches,  Zvi  and  Lichtenberg,  F.  "R&D  and  Productivity  Growth 
at  the  Firm  Level:  Is  There  Still  a  Relationship?"  in  R&D, 
Patents  and  Productivity,  ed. :  Zvi  Griliches.  Chicago: 
University  of  Chicago  Press,  1984,  pp.  465-496. 

Griliches,  Zvi.  "Productivity,  R&D  and  Basic  Research  at  the  Firm 
Level  in  the  1970' s,"  American  Economic  Review,  vol.  76,  no. 
1,  March  1986,  pp.  141-153. 

Griliches,  Zvi.  "Patents:  Recent  Trends  and  Puzzles,"  Brookings 
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Papers:  Microeconomics^  1989,  pp.  291-330. 


Levin,  Richard.  "Appropriability,  R&D  Spending  and  Technological 
Performance, "  American  Economic  Review  Papers  and 
Proceedings,  vol.  78,  no.  2,  May  1988,  pp.  424-428. 

Mansfield,  Edwin.  "Rates  of  Return  from  Industrial  R&D,"  American 
Economic  Review  Papers  and  Proceedings,  vol.  55,  no.  2,  May 
1965,  pp.  310-322. 

Mansfield,  Edwin.  Industrial  Research  and  Technological 

Innovation:  An  Econometric  Analysis.  New  York:  Norton  for 
the  Cowles  Foundation  for  Research  in  Economics,  Yale 
University,  1968. 

Mansfield,  Edwin;  Rapoport,  John;  Romeo,  Anthony;  Wagner,  Samuel 
and  Beardsley,  George.  "Social  and  Private  Rates  of  Return 
from  Industrial  Innovations,"  Quarterly  Journal  of  Economics, 
vol.  91,  no.  2,  May  1977,  pp.  221-240. 

Terleckyj,  N.  E.  Effects  of  R&D  on  the  Productivity  Growth  of 
Industries:  An  Exploratory  Study.  Washington,  D.C.:  National 
Planning  Association,  1974. 

Terleckyj,  N.E.  "Direct  and  Indirect  Effects  of  Industrial 
Research  and  Development  on  the  Productivity  Growth  of 
Industries,"  in  New  Developments  in  Productivity  Meas\irement 
and  Analysis,  eds.:  J.  Kendrick  and  B.  Vaccara.  Chicago: 
University  of  Chicago  Press,  1984,  pp.  359-386. 

Solow,  Robert  M.  "Technical  Change  and  the  Aggregate  Production 
Function,"  Review  of  Economics  and  Statistics,  vol.  39,  no.  3 
August  1957,  pp.  312-320. 

Case  Studies 

Bate 1 le .  Interactions  of  Science  and  Technology  in  the  Innovation 
Process:  Some  Case  Studies.  Prepared  for  the  National 
Science  Foundation,  Columbus,  Ohio:  Batelle  Columbus 

Laboratories,  1973. 

Charles  River  Associates.  Productivity  Impacts  of  NBS  R&D:  A  Case 
Study  of  the  NBS  Semiconductor  Technology  Program. 

Gaithersburg,  MD:  National  Bureau  of  Standards,  1981. 

Institute  of  Defense  Analysis  (IDA) .  "DARPA  Technical 

Accomplishments,"  vols.  1-3,  IDA  paper  p-2192  (v.l),  IDA 
paper  p-2429  (v.2),  IDA  paper  p-2538  (v.3),  Alexandria,  VA: 
IDA,  1990,  1991,  1992. 

ITT  Research  Institute.  Technology  in  Retrospect  and  Critical 
Events  in  Science.  Washington,  D.C.:  National  Science 

Foundation,  1968. 

Rosenberg,  Nathan.  "A  Historical  Overview  of  the  Evolution  of 
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Federal  Investment  in  R&D  Since  World  War  II/"  paper 
presented  at  the  workshop/  "The  Federal  Role  in  Research 
and  Development/"  Washington/  D.C.:  November  21-22/  1985. 

Sherwin/  C.W.  and  Isensen,  R.S.  First  Interim  Report  on  Project 
HINDSIGHT  (Summary) .  Washington/  D.C.:  Office  of  the  Director 
of  Defense  Research  and  Engineering/  1966. 

Critiques : 

Industry 

Kleinknecht/  Alfred  and  Bain/  Donald/  eds.  New  Concepts  in 
Innovation  Output  Measurements.  New  York:  St.  Martin's  Press, 
1993. 

McLoughlin,  G.J.  SEMATECH:  Issues  in  Evaluation  and  Assessment. 
Washington/  D.C.:  Congressional  Research  Service/  Science 
Policy  Research  Division/  #92-749SPR/  1992. 

Sorensen/  David  P.;  Nelson/  Kerry  S.  and  Tomsyck,  John  P. 
"Industrial  R&D  Program  Evaluation  Techniques,"  Evaluation 
Review,  vol.  18/  no.  1,  February  1994,  pp.  52-64. 

Econometric  Analysis 

GrilicheS/  Zvi.  "Issues  in  Assessing  the  Contribution  of  R&D  to 
Productivity  Growth,"  Bell  Journal  of  Economics,  vol.  10,  no. 
1/  1979/  pp.  92-116. 

GrilicheS/  Zvi.  "Patent  Statistics  as  Economic  Indicators;  A 
Survey,"  Journal  of  Economic  Literature,  vol.  28,  December 
1990/  pp.  1661-1707. 

Mansfield,  Edwin.  "R&D  and  Innovation;  Some  Empirical  Findings." 
in  R&D.  Patents,  and  Productivity,  ed. ;  Zvi  Griliches. 
Chicago:  The  University  of  Chicago  Press,  1984,  pp.  127-157. 

Mansfield,  Edwin.  "Social  Returns  from  R&D:  Findings,  Methods,  and 

Limitations,"  Research _ Technology  Management. 

November/December  1991,  pp.  24-27. 

Narin,  Francis;  Olivastro,  Dominic  and  Stevens,  Kimberly  A. 
"Bibliometrics/Theory,  Practice,  and  Problems,"  Evaluation 
Review,  vol.  18,  no.  1,  February  1994,  pp.  65-76. 

Casa  Studies 

Irvine,  John  and  Martin,  Ben.  Foresight  in  Science.  Picking  the 
Winners .  London:  Frances  Pinter,  1984,  chapter  2. 

Kostoff,  Ronald  N.  "Assessing  Research  Impact:  Semiquantitative 
Methods,"  Evaluation  Review,  vol.  18,  no.  1,  February  1994, 
pp.  11-19. 

Office  of  Technology  Investment.  Research  Funding  as  an 
Investment:  Can  We  Measure  The  Returns?  Report  for  Committee 
on  Science  and  Technology,  u.S.  House  of  Representatives, 
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Washington/  D.C.:  U.S.  Government  Printing  Office/  1986. 

Wagner/  Caroline/  and  Flanagan/  Ann.  "Rand  Workshop  on  the  Metrics 
of  Fundamental  Science:  A  Summary/"  prepared  for  the  Office  of 
Science  and  Technology  Policy/  February  1995. 


II.  Tailoring  Metrics  To  Economic  Policy  Objectives:  The  Case  of 
ATP.  MEP.  and  the  NIST  Labs 
a.  NIST's  Approach  to  Metrics 

Examples : 

HaineS/  Ruth.  "Project  Reporting  and  Evaluation:  NIST  Manufacturing 
Extension  Partnership."  December  1993. 

Link/  A.N.  Advanced  Technology  Program:  Economic  Study  of  the 

Printed  Wiring  Board  Venture  After  Two  Years.  Gaithersburg/ 
MD:  National  Institute  of  Standards  and  Technology/  April 
1993. 


Link/  A.N.  Economic  Impact  of  NIST  Research  on  Electromagnetic 
Interference.  Gaithersburg/  MD:  National  Institute  of 
Standards  and  Technology/  1991. 

Link/  A.N.  Economic  Impacts  of  NIST-Supported  Standards  for  the 
U.S.  Optical  Fiber  Industry:  1981-Present.  Gaithersburg/ 

MD:  National  Institute  of  Standards  and  Technology/  1991. 

Link/  A.N.  Economic  Impact  on  the  U.S.  Semiconductor  Industry  of 
NIST  Research  in  Electromiaration.  Gaithersburg/  MD:  National 
Institute  of  Standards  and  Technology/  1991. 

Manufacturing  Technology  Centers.  "Quarterly  Reporting 

Guidelines/"  Gaithersburg/  MD:  National  Institute  of 
Standards  and  Technology/  1993. 

National  Institute  of  Standards  and  Technology.  Setting 
Priorities  and  Measuring  Results  at  the  National  Institute  of 
Standards  and  Technology.  Gaithersburg/  MD:  National 
Institute  of  Standards  and  Technology/  January  1994. 

Tassey/  Gregory.  "The  Rates  of  Return  to  Private  and  Public 

Investment  in  Technology/ "  paper  prepared  for  the  National 
Institute  of  Standards  and  Technology/  March  1995. 

U.S.  Department  of  Commerce/  National  Institute  of  Standards  and 
Technology/  Visiting  Committee  on  Advanced  Technology/  The 
Manufacturing  Technology  Centers  Program.  October  1990. 

Critiques : 

Office  of  Technology  Assessment .  Contributions  of  DOE  Weapons  Labs 
and  NIST  to  Semiconductor  Technology.  Washington/  D.C.  (OTA- 
ITE-585) /  September  1993. 


C-6 


Luria,  Dan.  A  Framework  for  Evaluating  the  NIST/MTC^s:  A  Summary 
Based  on  the  Recommendations  of  the  NIST/MTC  Evaluation 
Working  Group,  Ann  Arbor,  MI :  Industrial  Technology  Institute 
and  Midwest  Manufacturing  Technology  Center,  1993. 

Solomon  Associates.  "The  Advanced  Technology  Program,  An 
Assessment  of  Short  Term  Impacts:  First  Competition 
Associates."  Submitted  to  the  Advanced  Technology  Program, 
National  Institute  of  Standards  and  Technology,  February 
1993. 

Visiting  Committee  on  Advanced  Technology  and  National  Institute  of 
Standards  and  Technology.  NIST  R&D  100  Awards:  A  Technology 
Transfer  Study.  Gaithersburg,  MD:  National  Institute  for 
Standards  and  Technology,  July  1991. 

III.  Metrics  for  Multiple  Policy  Objectives:  The  Case  cf  TRp  anH 

Dual  Use 

a.  The  Use  of  Mehrics  for  Dual— Use  Investments 

Dual-Use  Studies: 

Adams,  M.  and  Spann,  M.S.  Technology  Commercialization  and 
Strategic  Reduction  by  Federal  Defense  and  Soece  Contractors. 
Huntsville,  AL:  Management  and  Marketing  Department,  The 
University  of  Alabama  at  Huntsville,  October  1991. 

Advanced  Research  Projects  Agency,  Program  Information  Package  for 
Defense  Technology  Conversion,  Reinvestment,  and  Transition 
Assistance.  March  1993. 

An  Assessment  of  the  Impact  of  the  Department  of  Defense  Very  High 
^eed  Integrated  Circuit  Program.  National  Materials  Advisory 
Board  Report  NMAB-382.  Washington,  D.C.:  National  Research 
Council,  January  1982. 

Beyer,  John.  Net  Rates  of  Return  on  Innovations.  Executive 

Summary .  Washington,  D.C.:  National  Science  Foundation, 
October  1978. 

Beyer,  John.  Net  Rates  of  Return  on  Innovations.  Volume  1. 

Technical — Report .  Washington,  D.C.:  National  Science 

Foundation,  October  1978. 

Beyer,  John.  Net  Rates  of  Return  on  Innovations.  Volume  II.  Cast* 
Studies .  Washington,  D.C.:  National  Science  Foundation, 
October  1978. 

Brown,  Harold,  et  al.  Critial  Issues  in  Defense  Conversion. 

Washington,  D.C.:  The  Center  for  Strategic  and  International 
Studies,  1994. 

Carnegie  Commission.  Organizing  the  Executive  Branch  for  a 
Stronger  National  Technology  Base.  September  1991. 
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Defense  Conversion  Commission.  Adjusting  to  the  Drawdown. 
December  1992. 

Department  of  Defense,  Office  of  the  Undersecretary  of  Defense  for 
Acquisition.  Very  High  Speed  Integrated  Circuit  (VHSIC)  - 
Final  Program  Report,  1980-1990.  Washington,  D.C.:  September 
1990. 

National  Center  for  Advanced  Technologies,  Multi-Use  Manufacturing. 
White  Paper,  1993. 

National  Economic  Council,  National  Security  Council,  and  Office  of 
Science  and  Technology  Policy.  Second  to  None:  Presrving 
Americans  Military  Advantage  Through  Dual-Use  Technology. 
February  1995. 

Office  of  Technology  Assessment.  Building  Future  Security; 
Strategies  for  Restructuring  the  Defense  Technology  and 
Industrial  Base.  Washington,  D.C.:  U.S.  Government  Printing 
Office,  1992 . 

Office  of  Technology  Assessment.  Redesigning  Defense:  Planning  the 
Transition  to  the  Future  U.S.  Defense  Industrial  Base. 
Washington,  D.C.:  U.S.  Government  Printing  Office,  1991. 

Tewksbury,  J.G.;  Crandall,  M.  S.;  and  Crane,  W.E.  A  Survey  on  the 
Net  Returns  on  Innovations.  Volume  1  -  Introduction  and 

Summary,  National  Science  Foundation,  May  1978. 

Tewksbury,  J.G.;  Crandall,  M.  S.;  and  Crane,  W.E.  A  Survey  on  the 
Net  Returns  on  Innovations,  Volume  2  -  Summary  and  Analysis. 
National  Science  Foundation,  May  1978. 

Tewksbury,  J.G.;  Crandall,  M.  S.;  and  Crane,  W.E.  A  Survey  on  the 
Net  Returns  on  Innovations,  Volume  3  -  Detailed  Case  Studies. 
National  Science  Foundation,  May  1978. 

Dual-Use  Policy  Critiques: 

Alic,  John  A.;  Branscomb,  Lewis  M.;  Brooks,  Harvey;  Carter,  Ashton 
B.  and  Epstein,  Gerald  L.  Bevond  Spinoff:  Military  and 
Commercial  Technologies  in  a  Changing  World.  Boston:  Harvard 
Business  School  Press,  1992. 

Bischak,  Gregory  A.  Towards  a  Peace  Economy  in  the  United  states; 

Essays _ on  Military  Industry,  Disarmament  and  Economic 

Conversion.  New  York:  St.  Martin's  Press,  1991. 

Berkowitz,  Bruce  D.  "Can  Defense  Research  Revive  U.S.  Industry?" 
Issues  in  Science  and  Technology.  Winter  1992-1993,  pp.  73-81. 

Berkowitz,  Bruce  D.  "Why  Defense  Reinvestment  Won't  Work," 
Technology  Review.  July  1994. 

Bozeman,  B.  and  Coker,  K.  "Assessing  the  Effectiveness  of 
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Technology  Transfer  from  U.S.  R&D  Laboratories:  The  Impact  of 
Market  Orientation,"  Technovation.  vol.  12,  no.  4,  May  1992, 
pp.  239-255. 

Branscomb,  Lewis  ed.  Empowering  Technology:  Implementina  a  U.S. 
Strategy.  Cambridge,  MA:  The  MIT  Press,  1993. 

Clarke,  Marianne  K.  and  Dobson,  Eric  M.,  Increasing  the 
Competitiveness  of  Americans  Manufacturers:  A  Review  of  State 
Industrial  Extension  Programs.  National  Governors 

Association. 

Dermer,  Jeffry,  ed.  Competitiveness  Through  Technology:  What 

Business  Needs  from  Government.  Lexington,  MA:  Lexington 
Books,  1986. 

Edson,  L.  "Washington's  Energizer,"  Across  the  Board,  vol.  29, 
nos.  1-2,  January /February  1992,  pp.  23-28. 

Fong,  Glenn  R.  "The  Potential  for  Industry  Policy:  Lessons  from 
the  Very  High  Speed  Integrated  Circuit  Program, "  Journal  of 
Policy  Analysis  and  Management,  vol.  5,  1986,  pp.  264-291. 

Fong,  Glenn  R.  "State  Strength,  Industry  Structure,  and  Industrial 
Policy:  American  and  Japanese  Experiences  in 

Microelectronics,"  Comparative  Politics,  vol.  22,  1990,  pp. 
273-299. 

Institute  for  Defense  Analyses.  Technology  Transfer  in  a  Changing 
National  Security  Enviroment.  Alexandria,  VA:  The  Institute, 
1990. 

Lall,  Betty  G.  and  Marlin,  John  Tapper,  et  al.  Building  a  Peanp 
Economy:  Opportunities  and  Problems  of  Post-Cold  War  Deff^nsp 
Cuts .  Boulder,  CO:  Westview  Press,  1992. 

Leoplod,  George.  "Critics  Hope  Planners  Apply  Lessons  from  VHSIC, " 
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